Oct 30 21:41:47.099411 ip-10-0-131-250 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:41:47.527563 ip-10-0-131-250 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:47.527563 ip-10-0-131-250 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:41:47.527563 ip-10-0-131-250 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:47.527563 ip-10-0-131-250 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:41:47.530999 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.530924 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.530999 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.530998 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.530999 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531002 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531006 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531010 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531015 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531018 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531021 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531024 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531027 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531030 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531032 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531035 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531037 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531040 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531042 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531045 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531047 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531050 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531053 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531057 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.531093 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531061 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531064 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531067 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531071 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531073 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531076 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531079 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531081 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531084 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531087 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531089 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531092 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531094 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531097 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531100 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531102 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531106 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531109 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531111 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531114 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.531542 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531116 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531119 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531121 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531124 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531127 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531130 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531132 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531135 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531137 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531140 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531142 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531144 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531147 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531149 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531153 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531156 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531158 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531161 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531163 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.532022 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531166 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531168 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531171 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531174 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531176 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531179 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531181 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531184 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531186 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531189 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531191 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531193 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531196 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531199 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531201 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531204 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531206 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531209 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531212 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.532459 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531215 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531218 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531220 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531223 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531225 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531228 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531231 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531233 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531236 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531239 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531241 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531243 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531246 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531248 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531251 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531253 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531256 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531258 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531261 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531263 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.532922 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531266 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531268 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531272 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531274 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531278 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531280 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.531283 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532038 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532045 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532047 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532051 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532054 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532058 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532061 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532063 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532066 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532068 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532071 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532074 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.533418 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532077 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532079 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532082 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532085 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532087 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532090 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532092 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532094 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532097 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532099 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532102 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532106 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532110 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532113 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532117 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532120 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532123 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532125 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532128 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.533905 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532130 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532133 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532135 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532138 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532140 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532143 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532146 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532149 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532152 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532155 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532158 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532160 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532163 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532165 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532168 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532172 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532176 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532179 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532182 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532185 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.534360 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532188 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532190 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532193 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532196 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532198 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532201 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532203 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532206 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532211 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532214 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532216 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532219 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532221 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532224 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532226 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532229 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532232 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532235 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532237 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.534856 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532240 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532243 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532245 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532248 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532250 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532253 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532256 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532259 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532261 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532264 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532266 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532269 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532271 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532274 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532277 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532279 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532282 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532284 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532287 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.535321 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532289 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532291 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532294 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532297 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532300 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532302 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532305 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532308 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532310 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532313 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532315 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532318 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532321 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532323 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532326 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532328 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532813 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532819 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532822 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.535814 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532825 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532828 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532830 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532833 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532835 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532838 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532841 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532843 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532846 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532848 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532851 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532854 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532859 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532861 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532864 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532866 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532869 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532872 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532874 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532877 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.536274 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532880 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532882 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532885 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532887 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532890 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532892 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532895 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532898 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532900 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532902 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532905 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532907 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532910 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532913 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532915 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532918 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532920 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532923 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532926 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.536777 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532930 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532934 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532937 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532940 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532943 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532946 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532951 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532954 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532959 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532962 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532964 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532967 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532970 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532972 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532974 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532977 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532979 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532982 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.537279 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532984 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532987 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532990 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532992 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532995 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.532997 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533000 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533002 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533005 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533007 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533010 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533012 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533014 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533017 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533019 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533022 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533024 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533028 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533031 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533033 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.537765 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533036 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533038 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533042 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533045 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533048 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533051 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533053 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533056 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533058 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533061 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533063 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533066 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533068 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533071 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533074 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533076 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533079 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533082 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533084 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.538248 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533087 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533090 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533092 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533095 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.533097 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534067 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534076 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534082 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534086 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534091 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534096 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534100 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534104 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534107 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534110 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534113 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534116 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534119 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534122 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534125 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534128 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534131 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534134 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:41:47.538745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534141 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534144 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534147 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534150 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534153 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534158 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534161 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534164 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534168 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534171 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534174 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534177 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534180 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534183 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534187 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534190 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534193 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534196 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534198 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534203 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534206 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534210 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534212 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534215 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:41:47.539275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534219 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534223 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534226 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534229 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534232 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534235 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534238 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534241 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534244 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534247 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534249 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534252 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534257 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534260 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534264 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534267 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534270 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534273 2579 flags.go:64] FLAG: --help="false" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534276 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534279 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534282 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534285 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534288 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:41:47.539867 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534292 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534295 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534298 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534300 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534305 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534308 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534311 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534313 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534316 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534319 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534322 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534325 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534328 2579 flags.go:64] FLAG: --lock-file="" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534330 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534333 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534336 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534341 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534344 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534347 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534350 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534353 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534356 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534359 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534362 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534368 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:41:47.540414 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534371 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534375 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534378 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534381 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534384 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534387 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534389 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534392 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534395 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534402 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534405 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534409 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534412 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534415 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534418 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534421 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534424 2579 flags.go:64] FLAG: --port="10250" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534427 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534430 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-001d43e3baee20010" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534432 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534435 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534438 2579 flags.go:64] FLAG: --register-node="true" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534441 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534445 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534448 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534451 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:41:47.541263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534453 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534457 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534460 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534463 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534466 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534471 2579 flags.go:64] FLAG: --runonce="false" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534474 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534477 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534480 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534483 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534486 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534489 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534492 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534495 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534498 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534501 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534504 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534508 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534511 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534515 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534517 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534523 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534525 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534528 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534532 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:41:47.542493 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534535 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534538 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534540 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534543 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534546 2579 flags.go:64] FLAG: --v="2" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534550 2579 flags.go:64] FLAG: --version="false" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534554 2579 flags.go:64] FLAG: --vmodule="" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534558 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.534561 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534695 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534699 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534703 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534708 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534711 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534727 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534730 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534733 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534735 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534738 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534740 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534743 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534746 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.543647 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534749 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534751 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534755 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534757 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534760 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534763 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534765 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534768 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534770 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534773 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534776 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534778 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534781 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534784 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534786 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534789 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534791 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534793 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534796 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534798 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.544609 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534801 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534803 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534807 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534809 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534812 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534815 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534817 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534820 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534823 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534825 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534828 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534830 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534833 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534835 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534839 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534841 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534844 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534846 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534849 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534851 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.545531 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534854 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534857 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534859 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534862 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534865 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534867 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534870 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534873 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534875 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534878 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534881 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534883 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534886 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534888 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534892 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534895 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534897 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534900 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534903 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534905 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.546378 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534908 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534910 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534913 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534915 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534919 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534923 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534926 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534929 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534932 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534934 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534937 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534939 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534942 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534944 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534947 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534952 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534954 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534957 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.546896 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534959 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534962 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534964 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534967 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534969 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534972 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534978 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534981 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534983 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534985 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534988 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534991 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.534995 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.535021 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:47.547419 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535056 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535061 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535064 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535067 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535070 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535073 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535076 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535079 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535082 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535084 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535087 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535089 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535092 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535095 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535097 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535100 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535102 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535105 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535107 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535110 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.547804 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535112 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535115 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535117 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535120 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535124 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535126 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535129 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535131 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535134 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535137 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535140 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535143 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535145 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535148 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535150 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535153 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535155 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535158 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535160 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535163 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.548508 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535165 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535168 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535170 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535173 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535175 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535178 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535180 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535183 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535185 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535188 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535190 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535193 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535196 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535200 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535202 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535205 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535209 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535211 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535214 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.549197 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535217 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535219 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535222 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535225 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535227 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535230 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535232 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535235 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535237 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535240 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535243 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535245 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535248 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535250 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535253 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535255 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535258 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535260 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535263 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.549989 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535265 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535268 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535272 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535275 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535278 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535280 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535283 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535286 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535288 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535291 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535294 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535297 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535299 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535302 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535304 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535308 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535311 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535313 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535316 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.550636 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535319 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535322 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535324 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535327 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535330 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535333 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535336 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535338 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.535341 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.543867 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.543886 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.543908 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.543916 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544026 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544035 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544041 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544045 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544052 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544058 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.551146 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544063 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544067 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544071 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544076 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544080 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544084 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544089 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544093 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544098 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544102 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544107 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544111 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544124 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544129 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544133 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544137 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544141 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544145 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544149 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.551604 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544154 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544158 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544163 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544168 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544172 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544177 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544181 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544185 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544189 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544194 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544198 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544201 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544206 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544210 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544215 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544219 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544223 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544227 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544231 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544236 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.552128 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544240 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544244 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544248 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544252 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544256 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544260 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544272 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544276 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544280 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544284 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544288 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544293 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544297 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544301 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544306 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544311 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544315 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544319 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544323 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544327 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.552602 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544332 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544336 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544340 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544346 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544353 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544358 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544363 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544367 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544372 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544376 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544380 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544385 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544389 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544393 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544397 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544401 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544405 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544409 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544413 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.553265 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544425 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544430 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544434 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544438 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544442 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544447 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544451 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544455 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544459 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544464 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544475 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544480 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544484 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544488 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544493 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544497 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544501 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544505 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544509 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544513 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.553812 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544517 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544522 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.544562 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544897 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544907 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544912 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544917 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544921 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544925 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544929 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544934 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544938 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544942 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544954 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544958 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544963 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.554309 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544967 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544971 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544976 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544980 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544985 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544989 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544994 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.544998 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545005 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545011 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545016 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545021 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545026 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545030 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545034 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545039 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545043 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545047 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545051 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.554815 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545055 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545060 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545064 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545068 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545073 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545079 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545084 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545089 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545093 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545097 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545101 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545113 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545118 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545122 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545126 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545130 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545135 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545140 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545144 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.555264 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545149 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545153 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545158 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545162 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545166 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545170 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545174 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545178 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545182 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545186 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545191 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545195 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545199 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545203 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545207 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545212 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545216 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545220 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545224 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545229 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.555888 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545233 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545237 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545241 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545245 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545249 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545262 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545267 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545271 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545276 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545280 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545284 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545288 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545292 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545296 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545300 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545305 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545309 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545313 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545317 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545321 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.556687 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545326 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545330 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545334 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545338 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545342 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545346 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545350 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545354 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545358 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545362 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545371 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545375 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545380 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545384 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545388 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.557204 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.545426 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545529 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545537 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545557 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545562 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545566 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545570 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545575 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545579 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545583 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545588 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545592 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545596 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545601 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545605 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545611 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545618 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545623 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545628 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:41:47.557625 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545632 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545636 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545640 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545645 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545649 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545653 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545657 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545662 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545666 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545673 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545677 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545682 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545686 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545690 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545694 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545698 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545702 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545730 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545735 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:41:47.558139 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545740 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545745 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545749 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545753 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545757 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545762 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545766 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545770 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545774 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545778 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545782 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545788 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545792 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545797 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545801 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545805 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545809 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545813 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545818 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545822 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545826 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:41:47.558829 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545830 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545836 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545840 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545844 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545848 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545853 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545857 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545861 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545865 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545870 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545886 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545890 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545894 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545898 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545902 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545907 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545911 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545915 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545919 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:41:47.559369 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545923 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545927 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545931 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545935 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545939 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545944 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545948 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545952 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545956 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545960 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545964 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545969 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545973 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545977 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545983 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545987 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545992 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.545996 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546000 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546004 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:41:47.559832 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546008 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546012 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546016 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546021 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546033 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546040 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546044 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546051 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:47.546066 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.546150 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.551653 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.552647 2579 server.go:1014] "Starting client certificate rotation" Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.552751 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.552794 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.554778 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:41:47.560359 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.555906 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:41:47.565193 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.565181 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:41:47.569021 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.569005 2579 log.go:25] "Validated CRI v1 image API" Oct 30 21:41:47.569894 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.569871 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:41:47.577697 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.577677 2579 fs.go:135] Filesystem UUIDs: map[0a34cad0-2997-4f5b-b98e-4101ebf0099a:/dev/nvme0n1p3 2f31de20-e1a4-47c3-932a-be00f043c906:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:41:47.577778 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.577697 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:41:47.583219 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.583108 2579 manager.go:217] Machine: {Timestamp:2026-10-30 21:41:47.581664843 +0000 UTC m=+0.354754047 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101520 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27aa86fca7ff3ca2009b398f0ba360 SystemUUID:ec27aa86-fca7-ff3c-a200-9b398f0ba360 BootID:cbee68c7-ed4c-45ec-abeb-0c55525242ee Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:df:88:16:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:df:88:16:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:b6:ae:74:2f:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:41:47.583219 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.583218 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:41:47.583308 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.583287 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:41:47.586375 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.586350 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:41:47.586504 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.586376 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-250.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:41:47.586544 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.586506 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:41:47.586544 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.586513 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:41:47.586608 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.586548 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:41:47.586608 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.586568 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:41:47.587704 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.587693 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:41:47.591983 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.591971 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:41:47.592106 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.592097 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:41:47.592900 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.592890 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:41:47.592936 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.592909 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:41:47.592936 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.592920 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:41:47.592936 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.592928 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:41:47.592936 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.592935 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:41:47.593470 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.593451 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:41:47.597585 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.597561 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:41:47.604089 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.604076 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:41:47.605005 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.604993 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:41:47.605044 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605023 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:41:47.605171 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605161 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:41:47.605201 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605178 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:41:47.605201 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605187 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:41:47.605201 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605195 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:41:47.605293 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605203 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:41:47.605293 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605211 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:41:47.605981 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605971 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:41:47.606012 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605982 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:41:47.606012 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605989 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:41:47.606012 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.605995 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:41:47.606012 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.606003 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:41:47.606012 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.606011 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:41:47.606797 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.606786 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:41:47.606831 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.606801 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:41:47.610605 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.610591 2579 server.go:1263] "Started kubelet" Oct 30 21:41:47.610765 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.610696 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:41:47.610911 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.610681 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:41:47.610971 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.610930 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:41:47.611224 ip-10-0-131-250 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:41:47.612468 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.612452 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:41:47.613561 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.613545 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:41:47.616753 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.616736 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:41:47.619209 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.619191 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:41:47.620452 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.620438 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:41:47.620629 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.620594 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:47.621469 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621454 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:41:47.621469 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621458 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:41:47.621594 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621479 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:41:47.621594 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621575 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:41:47.621594 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621585 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:41:47.621863 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621849 2579 factory.go:153] Registering CRI-O factory Oct 30 21:41:47.621939 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621867 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 21:41:47.621939 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621918 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:41:47.621939 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621927 2579 factory.go:55] Registering systemd factory Oct 30 21:41:47.621939 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621934 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:41:47.622109 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621951 2579 factory.go:103] Registering Raw factory Oct 30 21:41:47.622109 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.621964 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 21:41:47.623079 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.623059 2579 manager.go:319] Starting recovery of all containers Oct 30 21:41:47.634777 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.634756 2579 manager.go:324] Recovery completed Oct 30 21:41:47.639765 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.639750 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:47.641248 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.641233 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:47.641324 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.641265 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:47.641324 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.641280 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:47.641892 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.641879 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:41:47.641892 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.641888 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:47.641967 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.641892 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:41:47.641967 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.641913 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:41:47.642598 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.642582 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:47.642669 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.642608 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:47.642669 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.642618 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:47.644077 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.644066 2579 policy_none.go:50] "Start" Oct 30 21:41:47.644134 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.644080 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:41:47.644134 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.644090 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:41:47.645919 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.645906 2579 policy_none.go:44] "Start" Oct 30 21:41:47.681006 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.680844 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.688922 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.688945 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.688955 2579 server.go:87] "Starting device plugin registration server" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.689203 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.689216 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.689293 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.689382 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.689391 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.689829 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.689871 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.698923 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.700182 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.700199 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.700231 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:41:47.705862 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.700261 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:41:47.754640 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.753503 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.us-west-1.compute.internal.18e36c9f318d9feb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47.610554347 +0000 UTC m=+0.383643550,LastTimestamp:2026-10-30 21:41:47.610554347 +0000 UTC m=+0.383643550,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.us-west-1.compute.internal,}" Oct 30 21:41:47.790190 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.790143 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:47.790904 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.790887 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:47.790979 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.790920 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:47.790979 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.790937 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:47.790979 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.790962 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.800520 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.800498 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal"] Oct 30 21:41:47.800608 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.800558 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:47.801537 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.801515 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-250.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:41:47.801642 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.801532 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:47.801642 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.801571 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:47.801642 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.801585 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:47.801815 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.801563 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-250.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:41:47.803875 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.803859 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:47.804021 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.804072 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804035 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:47.804528 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804511 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:47.804600 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804536 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:47.804600 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804546 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:47.804600 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804517 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:47.804703 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804620 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:47.804703 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.804633 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:47.806758 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.806741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.806848 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.806765 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:47.807395 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.807376 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:47.807492 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.807400 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:47.807492 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.807409 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:47.822771 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.822754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.822843 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.822777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.822843 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.822796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/801ebfba1db02fef7c7a04b1978809fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"801ebfba1db02fef7c7a04b1978809fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.825112 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.825096 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.827033 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.827014 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.831038 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.831022 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:41:47.831106 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.831070 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxpd4" Oct 30 21:41:47.859673 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.859642 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:41:47.859778 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.859687 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-250.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:41:47.908991 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.908947 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 21:41:47.908991 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.908969 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:41:47.908991 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.908932 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-250.us-west-1.compute.internal.18e36c9f3361fb6d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-250.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47.641248621 +0000 UTC m=+0.414337835,LastTimestamp:2026-10-30 21:41:47.641248621 +0000 UTC m=+0.414337835,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.us-west-1.compute.internal,}" Oct 30 21:41:47.912773 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:47.912759 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-250.us-west-1.compute.internal\" not found" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.923018 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.922999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.923065 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.923043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.923115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.923070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.923115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.923087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/801ebfba1db02fef7c7a04b1978809fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"801ebfba1db02fef7c7a04b1978809fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.923115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.923088 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:47.923228 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:47.923133 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/801ebfba1db02fef7c7a04b1978809fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"801ebfba1db02fef7c7a04b1978809fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:48.103766 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.103697 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-250.us-west-1.compute.internal\" not found" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:48.113827 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.113808 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:48.114824 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.114807 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:48.114912 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.114837 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:48.114912 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.114854 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:48.114912 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.114882 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:48.129768 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.129751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:48.131921 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.131906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:48.172045 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.172030 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:48.172110 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.172052 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-250.us-west-1.compute.internal\": node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.280030 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.279922 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.381032 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.380976 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.481488 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.481468 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.552680 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.552665 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:41:48.553042 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.552799 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ad8669e46bc8f4e2bbe5ac45b704f8a7-5cb0796d801b0643.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-250.us-west-1.compute.internal.18e36c9f3361fb6d\": read tcp 10.0.131.250:60314->13.184.210.151:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-250.us-west-1.compute.internal.18e36c9f3361fb6d default 4808 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-250.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-250.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-250.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47 +0000 UTC,LastTimestamp:2026-10-30 21:41:47.801559627 +0000 UTC m=+0.574648834,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-250.us-west-1.compute.internal,}" Oct 30 21:41:48.581892 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.581877 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.682687 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.682649 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.783133 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.783113 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.837197 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.837170 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:49 +0000 UTC" deadline="2028-05-02 04:09:30.045142341 +0000 UTC" Oct 30 21:41:48.837197 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.837193 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13182h27m41.207952643s" Oct 30 21:41:48.883455 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.883437 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:48.900597 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.900584 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:41:48.901433 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.901418 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:41:48.901505 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.901447 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:41:48.901505 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:48.901462 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:41:48.983535 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:48.983491 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:49.072003 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.071979 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:49.084109 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:49.084087 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:49.184309 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:49.184273 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:41:49.223067 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.223043 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:49.263835 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:49.263786 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908bb8ba293781da3344be83dfda2377.slice/crio-80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9 WatchSource:0}: Error finding container 80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9: Status 404 returned error can't find the container with id 80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9 Oct 30 21:41:49.264172 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:49.264153 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801ebfba1db02fef7c7a04b1978809fc.slice/crio-4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce WatchSource:0}: Error finding container 4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce: Status 404 returned error can't find the container with id 4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce Oct 30 21:41:49.270385 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.270369 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:41:49.321179 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.321156 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:49.383004 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.382984 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:41:49.385604 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.385592 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:41:49.482212 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.482191 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:49.482345 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:49.482302 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:41:50.894472 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:50.894452 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:41:50.932348 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:50.932330 2579 apiserver.go:52] "Watching apiserver" Oct 30 21:41:50.958525 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:50.958508 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:41:51.034274 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.034258 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:41:51.034580 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.034559 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-vvgps","openshift-multus/network-metrics-daemon-g7hlg","openshift-network-diagnostics/network-check-target-sz4m5","kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k","openshift-cluster-node-tuning-operator/tuned-ct72s","openshift-dns/node-resolver-lq586","openshift-image-registry/node-ca-wqw28","openshift-multus/multus-additional-cni-plugins-jw58x","openshift-network-operator/iptables-alerter-59qkx","openshift-ovn-kubernetes/ovnkube-node-vgt94","kube-system/global-pull-secret-syncer-qdn6l","kube-system/konnectivity-agent-v4lbg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal"] Oct 30 21:41:51.038500 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.038471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.040564 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.040540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.042702 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.042683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:51.042810 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.042759 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:41:51.044928 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.044911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.047024 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.047010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.049109 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.049094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.051195 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.051177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.053356 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.053335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.055558 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.055545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:51.055646 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.055618 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:41:51.057723 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.057709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.060070 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.060055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.062301 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.062261 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"908bb8ba293781da3344be83dfda2377","Type":"ContainerStarted","Data":"80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9"} Oct 30 21:41:51.062392 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.062315 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"801ebfba1db02fef7c7a04b1978809fc","Type":"ContainerStarted","Data":"4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce"} Oct 30 21:41:51.062392 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.062327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.062484 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.062390 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:41:51.071038 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysconfig\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.071120 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-run\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.071120 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7m6w\" (UniqueName: \"kubernetes.io/projected/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-kube-api-access-w7m6w\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.071120 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-var-lib-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071215 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-konnectivity-ca\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.071215 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f22346-7f81-4d27-92b3-fde049d58d93-host\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.071215 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-modprobe-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.071320 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-tuned\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.071320 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-config\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071389 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-env-overrides\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071389 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhd6\" (UniqueName: \"kubernetes.io/projected/cde9eb3a-08fa-49c1-bd82-38b414b0235f-kube-api-access-5bhd6\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.071389 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-system-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.071389 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cnibin\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.071520 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.071520 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-kubelet\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071520 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071520 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-dbus\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.071520 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:51.071520 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-conf\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-multus-certs\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5n7q\" (UniqueName: \"kubernetes.io/projected/6a4f783d-df3d-4c74-939e-22bf246e9738-kube-api-access-w5n7q\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-netns\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-slash\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-node-log\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-bin\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-device-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75ql\" (UniqueName: \"kubernetes.io/projected/f0f22346-7f81-4d27-92b3-fde049d58d93-kube-api-access-m75ql\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6dfl\" (UniqueName: \"kubernetes.io/projected/7b5b5c38-04af-42ba-9845-41520e647cbe-kube-api-access-n6dfl\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-etc-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-ovn\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-netd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6780389-4c8f-449a-a007-8682974061cb-tmp-dir\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-hostroot\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:51.071947 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.071976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-agent-certs\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-cni-binary-copy\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-kubernetes\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-sys\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-host\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-systemd-units\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-log-socket\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-bin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-kubelet\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-conf-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072246 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-tmp\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-k8s-cni-cncf-io\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-netns\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjt2t\" (UniqueName: \"kubernetes.io/projected/df4f0be9-a14d-4120-a10f-c815830aef30-kube-api-access-kjt2t\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc6h\" (UniqueName: \"kubernetes.io/projected/b6780389-4c8f-449a-a007-8682974061cb-kube-api-access-spc6h\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-registration-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-sys-fs\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-cnibin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-daemon-config\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f22346-7f81-4d27-92b3-fde049d58d93-serviceca\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7bz\" (UniqueName: \"kubernetes.io/projected/c0a1d58f-d414-42c1-8505-79acb178f8cf-kube-api-access-xc7bz\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-multus\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-kubelet-config\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.072658 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cde9eb3a-08fa-49c1-bd82-38b414b0235f-iptables-alerter-script\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-socket-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-systemd\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-system-cni-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-systemd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb00f28-52e5-48e2-a034-7193a74f8157-ovn-node-metrics-cert\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-script-lib\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-var-lib-kubelet\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cde9eb3a-08fa-49c1-bd82-38b414b0235f-host-slash\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6780389-4c8f-449a-a007-8682974061cb-hosts-file\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-socket-dir-parent\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-lib-modules\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br58m\" (UniqueName: \"kubernetes.io/projected/fdb00f28-52e5-48e2-a034-7193a74f8157-kube-api-access-br58m\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.073182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.072866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-etc-kubernetes\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.095136 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.095120 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:41:51.095286 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.095273 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4c9l\"" Oct 30 21:41:51.095286 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.095282 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:41:51.095375 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.095309 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6vmmq\"" Oct 30 21:41:51.095757 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.095742 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:41:51.095931 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.095920 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:41:51.096605 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.096593 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:41:51.098277 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.098261 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:41:51.098409 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.098390 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:41:51.098472 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.098459 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:41:51.098525 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.098473 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rfsrz\"" Oct 30 21:41:51.100879 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.100864 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:41:51.100960 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.100949 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:41:51.101045 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.101033 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fxktb\"" Oct 30 21:41:51.101310 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.101299 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:41:51.103951 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.103936 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xw9dh\"" Oct 30 21:41:51.103994 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.103955 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:41:51.104024 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.104005 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:41:51.104024 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.104016 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:41:51.104090 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.104023 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:41:51.104090 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.104055 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:41:51.104090 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.104065 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:41:51.104090 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.104075 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpjmc\"" Oct 30 21:41:51.104090 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.104079 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:41:51.132965 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.132952 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:41:51.137381 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.137354 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:41:51.137428 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.137381 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:41:51.144652 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144628 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qdnct\"" Oct 30 21:41:51.144740 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144669 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:41:51.144740 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144678 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:41:51.144740 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144688 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:41:51.144740 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144692 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:41:51.144740 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144705 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:41:51.144740 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144713 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:41:51.144740 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144739 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qtzdm\"" Oct 30 21:41:51.144997 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144765 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:41:51.144997 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144792 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mk72l" Oct 30 21:41:51.144997 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.144800 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pvwl\"" Oct 30 21:41:51.160399 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.160382 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:41:51.174021 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-bin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174073 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-kubelet\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174073 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-conf-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174073 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.174073 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-tmp\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-k8s-cni-cncf-io\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-netns\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-kubelet\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjt2t\" (UniqueName: \"kubernetes.io/projected/df4f0be9-a14d-4120-a10f-c815830aef30-kube-api-access-kjt2t\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174151 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-bin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc6h\" (UniqueName: \"kubernetes.io/projected/b6780389-4c8f-449a-a007-8682974061cb-kube-api-access-spc6h\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-conf-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174175 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-netns\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-registration-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174206 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-k8s-cni-cncf-io\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-sys-fs\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174233 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-sys-fs\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174239 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174240 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-registration-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-cnibin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-daemon-config\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-cnibin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f22346-7f81-4d27-92b3-fde049d58d93-serviceca\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174377 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174430 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7bz\" (UniqueName: \"kubernetes.io/projected/c0a1d58f-d414-42c1-8505-79acb178f8cf-kube-api-access-xc7bz\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-multus\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-kubelet-config\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174569 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cde9eb3a-08fa-49c1-bd82-38b414b0235f-iptables-alerter-script\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174580 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-multus\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-socket-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-kubelet-config\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.174814 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-systemd\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174715 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-socket-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-system-cni-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174753 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-systemd\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-systemd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f22346-7f81-4d27-92b3-fde049d58d93-serviceca\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb00f28-52e5-48e2-a034-7193a74f8157-ovn-node-metrics-cert\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-system-cni-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-script-lib\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174872 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-systemd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-var-lib-kubelet\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cde9eb3a-08fa-49c1-bd82-38b414b0235f-host-slash\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-var-lib-kubelet\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6780389-4c8f-449a-a007-8682974061cb-hosts-file\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-socket-dir-parent\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.175815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174993 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cde9eb3a-08fa-49c1-bd82-38b414b0235f-host-slash\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.174896 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-daemon-config\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-lib-modules\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br58m\" (UniqueName: \"kubernetes.io/projected/fdb00f28-52e5-48e2-a034-7193a74f8157-kube-api-access-br58m\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175060 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-socket-dir-parent\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6780389-4c8f-449a-a007-8682974061cb-hosts-file\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-etc-kubernetes\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cde9eb3a-08fa-49c1-bd82-38b414b0235f-iptables-alerter-script\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175101 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysconfig\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-run\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7m6w\" (UniqueName: \"kubernetes.io/projected/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-kube-api-access-w7m6w\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-var-lib-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-konnectivity-ca\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f22346-7f81-4d27-92b3-fde049d58d93-host\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-modprobe-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-tuned\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.176672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-config\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-env-overrides\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhd6\" (UniqueName: \"kubernetes.io/projected/cde9eb3a-08fa-49c1-bd82-38b414b0235f-kube-api-access-5bhd6\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-system-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175546 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-modprobe-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175558 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-lib-modules\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysconfig\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cnibin\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-run\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-etc-kubernetes\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.175980 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-script-lib\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-kubelet\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-dbus\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:51.177425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-conf\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-multus-certs\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5n7q\" (UniqueName: \"kubernetes.io/projected/6a4f783d-df3d-4c74-939e-22bf246e9738-kube-api-access-w5n7q\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-netns\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-slash\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-node-log\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-bin\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176387 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cnibin\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176435 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-system-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-device-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75ql\" (UniqueName: \"kubernetes.io/projected/f0f22346-7f81-4d27-92b3-fde049d58d93-kube-api-access-m75ql\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6dfl\" (UniqueName: \"kubernetes.io/projected/7b5b5c38-04af-42ba-9845-41520e647cbe-kube-api-access-n6dfl\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-etc-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-ovn\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-netd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178115 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6780389-4c8f-449a-a007-8682974061cb-tmp-dir\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-hostroot\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-agent-certs\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-cni-binary-copy\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-kubernetes\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-sys\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.176999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-konnectivity-ca\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-host\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177061 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-kubelet\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-systemd-units\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177096 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-log-socket\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.178587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177144 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-netd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177149 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-conf\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177167 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-multus-certs\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177199 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f22346-7f81-4d27-92b3-fde049d58d93-host\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-log-socket\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-device-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177316 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-slash\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-etc-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177369 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-kubernetes\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-node-log\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177402 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-netns\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.177419 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177419 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-ovn\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177036 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177070 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-dbus\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.179194 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177540 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-sys\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.177582 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.677551641 +0000 UTC m=+3.112867369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-bin\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-host\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177656 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-hostroot\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177674 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-systemd-units\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.177777 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-tmp\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.177964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6780389-4c8f-449a-a007-8682974061cb-tmp-dir\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.178055 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret podName:5214b527-3150-40e4-9162-f712d602c83a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.678043251 +0000 UTC m=+3.113358970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret") pod "global-pull-secret-syncer-qdn6l" (UID: "5214b527-3150-40e4-9162-f712d602c83a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.178420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-var-lib-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.178675 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-env-overrides\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.178719 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-tuned\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.178762 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.178874 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb00f28-52e5-48e2-a034-7193a74f8157-ovn-node-metrics-cert\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.178982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-cni-binary-copy\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.179673 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.179045 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-config\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.180540 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.180523 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-agent-certs\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.241804 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.241780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc6h\" (UniqueName: \"kubernetes.io/projected/b6780389-4c8f-449a-a007-8682974061cb-kube-api-access-spc6h\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.241907 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.241886 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjt2t\" (UniqueName: \"kubernetes.io/projected/df4f0be9-a14d-4120-a10f-c815830aef30-kube-api-access-kjt2t\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.242042 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.242024 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7bz\" (UniqueName: \"kubernetes.io/projected/c0a1d58f-d414-42c1-8505-79acb178f8cf-kube-api-access-xc7bz\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:51.247070 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.247054 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:51.247154 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.247076 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:51.247154 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.247089 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8v7rp for pod openshift-network-diagnostics/network-check-target-sz4m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:51.247154 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.247149 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp podName:55e5bc9a-1f53-4a57-88a5-446df5df92f0 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.747134596 +0000 UTC m=+3.182450321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8v7rp" (UniqueName: "kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp") pod "network-check-target-sz4m5" (UID: "55e5bc9a-1f53-4a57-88a5-446df5df92f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:51.249366 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.249340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5n7q\" (UniqueName: \"kubernetes.io/projected/6a4f783d-df3d-4c74-939e-22bf246e9738-kube-api-access-w5n7q\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.249503 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.249475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhd6\" (UniqueName: \"kubernetes.io/projected/cde9eb3a-08fa-49c1-bd82-38b414b0235f-kube-api-access-5bhd6\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.250615 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.250597 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6dfl\" (UniqueName: \"kubernetes.io/projected/7b5b5c38-04af-42ba-9845-41520e647cbe-kube-api-access-n6dfl\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.251421 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.251403 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7m6w\" (UniqueName: \"kubernetes.io/projected/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-kube-api-access-w7m6w\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.253912 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.253893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br58m\" (UniqueName: \"kubernetes.io/projected/fdb00f28-52e5-48e2-a034-7193a74f8157-kube-api-access-br58m\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.255529 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.255510 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75ql\" (UniqueName: \"kubernetes.io/projected/f0f22346-7f81-4d27-92b3-fde049d58d93-kube-api-access-m75ql\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.350145 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.350123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:41:51.357275 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.357260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:51.359972 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.359953 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde9eb3a_08fa_49c1_bd82_38b414b0235f.slice/crio-1e9b2fdaabbdc1d28a20c477974601a9d6327be98dba7d1dc58fbe8f334834a1 WatchSource:0}: Error finding container 1e9b2fdaabbdc1d28a20c477974601a9d6327be98dba7d1dc58fbe8f334834a1: Status 404 returned error can't find the container with id 1e9b2fdaabbdc1d28a20c477974601a9d6327be98dba7d1dc58fbe8f334834a1 Oct 30 21:41:51.363557 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.363539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:41:51.365124 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.365105 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5d51c9_d860_45d8_b9c0_6e11fdd73ca9.slice/crio-258afb86fd7380338c27257dffdb7e9382ebe0a158a2371be326e3021a1f057d WatchSource:0}: Error finding container 258afb86fd7380338c27257dffdb7e9382ebe0a158a2371be326e3021a1f057d: Status 404 returned error can't find the container with id 258afb86fd7380338c27257dffdb7e9382ebe0a158a2371be326e3021a1f057d Oct 30 21:41:51.371980 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.371963 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f22346_7f81_4d27_92b3_fde049d58d93.slice/crio-e5b9473052eeb2b8a8b3ae42fb92d746b4c4deedaa0d67b26f2362130f42ecfc WatchSource:0}: Error finding container e5b9473052eeb2b8a8b3ae42fb92d746b4c4deedaa0d67b26f2362130f42ecfc: Status 404 returned error can't find the container with id e5b9473052eeb2b8a8b3ae42fb92d746b4c4deedaa0d67b26f2362130f42ecfc Oct 30 21:41:51.372938 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.372922 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:41:51.376058 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.376044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:41:51.382929 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.382912 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4f0be9_a14d_4120_a10f_c815830aef30.slice/crio-4aeeaa3fa7a1ea99ff08721f2b363e8c981ac7d066eb0e8f4f0b8f2231be9a86 WatchSource:0}: Error finding container 4aeeaa3fa7a1ea99ff08721f2b363e8c981ac7d066eb0e8f4f0b8f2231be9a86: Status 404 returned error can't find the container with id 4aeeaa3fa7a1ea99ff08721f2b363e8c981ac7d066eb0e8f4f0b8f2231be9a86 Oct 30 21:41:51.383551 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.383447 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5b5c38_04af_42ba_9845_41520e647cbe.slice/crio-cc7bcadc0d31fb08a1ccad4202a4a7f5181c0eed12118303245697579da594b4 WatchSource:0}: Error finding container cc7bcadc0d31fb08a1ccad4202a4a7f5181c0eed12118303245697579da594b4: Status 404 returned error can't find the container with id cc7bcadc0d31fb08a1ccad4202a4a7f5181c0eed12118303245697579da594b4 Oct 30 21:41:51.390892 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.390848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lq586" Oct 30 21:41:51.392301 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.392286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vvgps" Oct 30 21:41:51.396401 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.396382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:41:51.400427 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.400407 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6780389_4c8f_449a_a007_8682974061cb.slice/crio-395f2ead7c0dda3ecb0cbbcd857bfad1dd0aff5748136c3d1d4b9e5fba58cd48 WatchSource:0}: Error finding container 395f2ead7c0dda3ecb0cbbcd857bfad1dd0aff5748136c3d1d4b9e5fba58cd48: Status 404 returned error can't find the container with id 395f2ead7c0dda3ecb0cbbcd857bfad1dd0aff5748136c3d1d4b9e5fba58cd48 Oct 30 21:41:51.404282 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.404205 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4f783d_df3d_4c74_939e_22bf246e9738.slice/crio-0d5571934de525b4caa88fda71a31989531babf03a13f2c59f86850e95ca0e03 WatchSource:0}: Error finding container 0d5571934de525b4caa88fda71a31989531babf03a13f2c59f86850e95ca0e03: Status 404 returned error can't find the container with id 0d5571934de525b4caa88fda71a31989531babf03a13f2c59f86850e95ca0e03 Oct 30 21:41:51.406593 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.406579 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1538ab8_ca5c_4f56_b86f_1a0ae2f313a7.slice/crio-f3b912f34e393b6b3180d7ef5ffbc7d1310437a7a431f9f1cf15d661f3636f5b WatchSource:0}: Error finding container f3b912f34e393b6b3180d7ef5ffbc7d1310437a7a431f9f1cf15d661f3636f5b: Status 404 returned error can't find the container with id f3b912f34e393b6b3180d7ef5ffbc7d1310437a7a431f9f1cf15d661f3636f5b Oct 30 21:41:51.411935 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.411920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:41:51.420283 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:41:51.420261 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb00f28_52e5_48e2_a034_7193a74f8157.slice/crio-adcbbb90aec8bc038ff107c3fcf2c08ad57cc6b7b1e6f03b7af381b831499d9d WatchSource:0}: Error finding container adcbbb90aec8bc038ff107c3fcf2c08ad57cc6b7b1e6f03b7af381b831499d9d: Status 404 returned error can't find the container with id adcbbb90aec8bc038ff107c3fcf2c08ad57cc6b7b1e6f03b7af381b831499d9d Oct 30 21:41:51.681626 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.681557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:51.681626 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.681602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:51.681786 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.681696 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:51.681786 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.681745 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret podName:5214b527-3150-40e4-9162-f712d602c83a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:52.681727008 +0000 UTC m=+4.117042728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret") pod "global-pull-secret-syncer-qdn6l" (UID: "5214b527-3150-40e4-9162-f712d602c83a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:51.681865 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.681799 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:51.681865 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.681831 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:41:52.681822253 +0000 UTC m=+4.117137973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:51.782620 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:51.782586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:51.782802 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.782743 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:51.782802 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.782761 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:51.782802 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.782773 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8v7rp for pod openshift-network-diagnostics/network-check-target-sz4m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:51.782932 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:51.782825 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp podName:55e5bc9a-1f53-4a57-88a5-446df5df92f0 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:52.782808031 +0000 UTC m=+4.218123752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v7rp" (UniqueName: "kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp") pod "network-check-target-sz4m5" (UID: "55e5bc9a-1f53-4a57-88a5-446df5df92f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:52.046706 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.046617 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"adcbbb90aec8bc038ff107c3fcf2c08ad57cc6b7b1e6f03b7af381b831499d9d"} Oct 30 21:41:52.048153 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.048127 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerStarted","Data":"f3b912f34e393b6b3180d7ef5ffbc7d1310437a7a431f9f1cf15d661f3636f5b"} Oct 30 21:41:52.049378 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.049354 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vvgps" event={"ID":"6a4f783d-df3d-4c74-939e-22bf246e9738","Type":"ContainerStarted","Data":"0d5571934de525b4caa88fda71a31989531babf03a13f2c59f86850e95ca0e03"} Oct 30 21:41:52.050644 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.050616 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ct72s" event={"ID":"7b5b5c38-04af-42ba-9845-41520e647cbe","Type":"ContainerStarted","Data":"cc7bcadc0d31fb08a1ccad4202a4a7f5181c0eed12118303245697579da594b4"} Oct 30 21:41:52.051593 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.051551 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4lbg" event={"ID":"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9","Type":"ContainerStarted","Data":"258afb86fd7380338c27257dffdb7e9382ebe0a158a2371be326e3021a1f057d"} Oct 30 21:41:52.052977 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.052953 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq586" event={"ID":"b6780389-4c8f-449a-a007-8682974061cb","Type":"ContainerStarted","Data":"395f2ead7c0dda3ecb0cbbcd857bfad1dd0aff5748136c3d1d4b9e5fba58cd48"} Oct 30 21:41:52.054304 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.054272 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" event={"ID":"df4f0be9-a14d-4120-a10f-c815830aef30","Type":"ContainerStarted","Data":"4aeeaa3fa7a1ea99ff08721f2b363e8c981ac7d066eb0e8f4f0b8f2231be9a86"} Oct 30 21:41:52.055534 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.055513 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wqw28" event={"ID":"f0f22346-7f81-4d27-92b3-fde049d58d93","Type":"ContainerStarted","Data":"e5b9473052eeb2b8a8b3ae42fb92d746b4c4deedaa0d67b26f2362130f42ecfc"} Oct 30 21:41:52.056778 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.056732 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59qkx" event={"ID":"cde9eb3a-08fa-49c1-bd82-38b414b0235f","Type":"ContainerStarted","Data":"1e9b2fdaabbdc1d28a20c477974601a9d6327be98dba7d1dc58fbe8f334834a1"} Oct 30 21:41:52.148283 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.148241 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:51 +0000 UTC" deadline="2028-04-25 14:30:06.601140139 +0000 UTC" Oct 30 21:41:52.148283 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.148282 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13024h48m14.452863478s" Oct 30 21:41:52.690727 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.690696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:52.690899 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.690762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:52.690976 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.690902 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:52.690976 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.690954 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret podName:5214b527-3150-40e4-9162-f712d602c83a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:54.690937093 +0000 UTC m=+6.126252801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret") pod "global-pull-secret-syncer-qdn6l" (UID: "5214b527-3150-40e4-9162-f712d602c83a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:52.691086 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.691017 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:52.691086 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.691051 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:41:54.691039086 +0000 UTC m=+6.126354795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:52.792056 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:52.791739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:52.792056 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.791890 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:52.792056 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.791911 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:52.792056 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.791924 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8v7rp for pod openshift-network-diagnostics/network-check-target-sz4m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:52.792056 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:52.791976 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp podName:55e5bc9a-1f53-4a57-88a5-446df5df92f0 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:54.79195928 +0000 UTC m=+6.227274987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v7rp" (UniqueName: "kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp") pod "network-check-target-sz4m5" (UID: "55e5bc9a-1f53-4a57-88a5-446df5df92f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:53.040621 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:53.039796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:53.040621 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:53.039925 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:41:53.040621 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:53.040020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:53.040621 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:53.040102 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:41:53.040621 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:53.040156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:53.040621 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:53.040218 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:41:53.148807 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:53.148742 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:51 +0000 UTC" deadline="2028-03-31 23:23:19.388514705 +0000 UTC" Oct 30 21:41:53.148807 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:53.148806 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12433h41m26.239713162s" Oct 30 21:41:54.712815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:54.712626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:54.713225 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:54.712867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:54.713225 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.712781 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:54.713225 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.712931 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:41:58.712915759 +0000 UTC m=+10.148231469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:54.713225 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.713058 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:54.713225 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.713119 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret podName:5214b527-3150-40e4-9162-f712d602c83a nodeName:}" failed. No retries permitted until 2026-10-30 21:41:58.713104148 +0000 UTC m=+10.148419857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret") pod "global-pull-secret-syncer-qdn6l" (UID: "5214b527-3150-40e4-9162-f712d602c83a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:54.813602 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:54.813563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:54.813778 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.813751 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:54.813778 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.813777 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:54.813916 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.813790 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8v7rp for pod openshift-network-diagnostics/network-check-target-sz4m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:54.813916 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:54.813846 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp podName:55e5bc9a-1f53-4a57-88a5-446df5df92f0 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:58.813827869 +0000 UTC m=+10.249143578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v7rp" (UniqueName: "kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp") pod "network-check-target-sz4m5" (UID: "55e5bc9a-1f53-4a57-88a5-446df5df92f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:55.040686 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:55.040645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:55.040792 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:55.040769 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:41:55.040841 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:55.040827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:55.040915 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:55.040896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:41:55.041067 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:55.041040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:55.041144 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:55.041126 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:41:55.082622 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:55.082574 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerStarted","Data":"a3a6fcb9c6db2698ddd31767bf5876bb66a256d0d36aef739c1c7b771e2b67e9"} Oct 30 21:41:55.091952 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:55.091926 2579 generic.go:356] "Generic (PLEG): container finished" podID="908bb8ba293781da3344be83dfda2377" containerID="4d5fb9e646c4d45e7da2f1facf040038d008dd5a78817c729a7bb4c63b260ac3" exitCode=0 Oct 30 21:41:55.092051 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:55.092026 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"908bb8ba293781da3344be83dfda2377","Type":"ContainerDied","Data":"4d5fb9e646c4d45e7da2f1facf040038d008dd5a78817c729a7bb4c63b260ac3"} Oct 30 21:41:55.106301 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:55.106274 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"801ebfba1db02fef7c7a04b1978809fc","Type":"ContainerStarted","Data":"9548578180e75180dae09454eb30122c6f8e5135becc891708f42ce6c9913517"} Oct 30 21:41:56.118099 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:56.117129 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="a3a6fcb9c6db2698ddd31767bf5876bb66a256d0d36aef739c1c7b771e2b67e9" exitCode=0 Oct 30 21:41:56.118099 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:56.117216 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerDied","Data":"a3a6fcb9c6db2698ddd31767bf5876bb66a256d0d36aef739c1c7b771e2b67e9"} Oct 30 21:41:56.132371 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:56.132313 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"908bb8ba293781da3344be83dfda2377","Type":"ContainerStarted","Data":"daeda2f5146714cbe1ba56576ec6933ee9af2036ff890a11c9d25f0fa334559c"} Oct 30 21:41:56.237547 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:56.237441 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" podStartSLOduration=6.237421427 podStartE2EDuration="6.237421427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.444948519 +0000 UTC m=+6.880264245" watchObservedRunningTime="2026-10-30 21:41:56.237421427 +0000 UTC m=+7.672737162" Oct 30 21:41:57.041450 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:57.040705 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:57.041450 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:57.040736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:57.041450 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:57.040807 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:41:57.041450 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:57.040862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:57.041450 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:57.040926 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:41:57.041450 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:57.040976 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:41:57.142603 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:57.142550 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4lbg" event={"ID":"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9","Type":"ContainerStarted","Data":"db0b83b95af0b8c664e8a45ff88bfd43c8892f90458f4eac249ccc82762b87dd"} Oct 30 21:41:57.257688 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:57.256513 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" podStartSLOduration=7.2564790089999995 podStartE2EDuration="7.256479009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:56.352080723 +0000 UTC m=+7.787396453" watchObservedRunningTime="2026-10-30 21:41:57.256479009 +0000 UTC m=+8.691794739" Oct 30 21:41:58.147598 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.147459 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" event={"ID":"df4f0be9-a14d-4120-a10f-c815830aef30","Type":"ContainerStarted","Data":"430a8265524480c96d7af6b7d0ce2df87fda1e12debba4beae5432cbbc1e48a2"} Oct 30 21:41:58.152659 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.152568 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wqw28" event={"ID":"f0f22346-7f81-4d27-92b3-fde049d58d93","Type":"ContainerStarted","Data":"0b490b04ef753db92ed96fd4501e706abf336cf3788e23b49e33ab184510d7e5"} Oct 30 21:41:58.268134 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.267150 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v4lbg" podStartSLOduration=4.658380685 podStartE2EDuration="9.267132633s" totalImagesPullingTime="4.608751948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:57.257634304 +0000 UTC m=+8.692950032" watchObservedRunningTime="2026-10-30 21:41:58.267132633 +0000 UTC m=+9.702448364" Oct 30 21:41:58.749016 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.748972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:58.749210 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.749052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:58.749274 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.749209 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:58.749274 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.749262 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret podName:5214b527-3150-40e4-9162-f712d602c83a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:06.749248067 +0000 UTC m=+18.184563772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret") pod "global-pull-secret-syncer-qdn6l" (UID: "5214b527-3150-40e4-9162-f712d602c83a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:41:58.749672 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.749652 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:58.749771 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.749705 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:42:06.749694329 +0000 UTC m=+18.185010035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:41:58.850032 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.849993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:58.850238 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.850218 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:41:58.850333 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.850249 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:41:58.850333 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.850263 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8v7rp for pod openshift-network-diagnostics/network-check-target-sz4m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:58.850333 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:58.850325 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp podName:55e5bc9a-1f53-4a57-88a5-446df5df92f0 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:06.850306826 +0000 UTC m=+18.285622553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v7rp" (UniqueName: "kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp") pod "network-check-target-sz4m5" (UID: "55e5bc9a-1f53-4a57-88a5-446df5df92f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:41:58.944737 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.944583 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:58.945133 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:58.945108 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:41:59.039556 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:59.038163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:41:59.039556 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:59.038286 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:41:59.039556 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:59.038697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:41:59.039556 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:59.038797 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:41:59.039556 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:59.038862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:41:59.039556 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:41:59.038944 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:41:59.064641 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:59.062334 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wqw28" podStartSLOduration=4.47727123 podStartE2EDuration="10.062315705s" totalImagesPullingTime="5.585044475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:58.268099241 +0000 UTC m=+9.703414969" watchObservedRunningTime="2026-10-30 21:41:59.062315705 +0000 UTC m=+10.497631433" Oct 30 21:41:59.751666 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:41:59.751641 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:42:00.037081 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.037001 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:41:59.751657676Z","UUID":"d9846300-d587-4e77-b17f-abbec49e81cc","Handler":null,"Name":"","Endpoint":""} Oct 30 21:42:00.038833 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.038413 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:42:00.038833 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.038453 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:42:00.162458 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.162386 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" event={"ID":"df4f0be9-a14d-4120-a10f-c815830aef30","Type":"ContainerStarted","Data":"af4e60ef62b1a77c5fedbbe8d8c037d85e5700b01b4894336d9a6833b7c82ab8"} Oct 30 21:42:00.164144 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.164111 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vvgps" event={"ID":"6a4f783d-df3d-4c74-939e-22bf246e9738","Type":"ContainerStarted","Data":"ce63149574dba1f28ffc694da3d0694b3ddf607dc22332e0245987c9010327fa"} Oct 30 21:42:00.167625 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.167603 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:00.168265 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.168245 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lq586" event={"ID":"b6780389-4c8f-449a-a007-8682974061cb","Type":"ContainerStarted","Data":"7573c3f4031d8bea49421c5b4c2bb4ed88998c90ae00b2f2de01333f252a6733"} Oct 30 21:42:00.284983 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:00.284937 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vvgps" podStartSLOduration=3.09667377 podStartE2EDuration="11.284925918s" totalImagesPullingTime="8.188252148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:00.284821147 +0000 UTC m=+11.720136877" watchObservedRunningTime="2026-10-30 21:42:00.284925918 +0000 UTC m=+11.720241643" Oct 30 21:42:01.039001 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:01.038761 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:01.039437 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:01.038840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:01.039437 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:01.039139 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:01.039437 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:01.039214 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:01.039437 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:01.038847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:01.039437 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:01.039348 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:01.173400 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:01.173276 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="4148184c5e12347eabbc7315648e2a7ae9be59efc8f227de38951a93727e0953" exitCode=0 Oct 30 21:42:01.173400 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:01.173328 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerDied","Data":"4148184c5e12347eabbc7315648e2a7ae9be59efc8f227de38951a93727e0953"} Oct 30 21:42:01.379632 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:01.379577 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-lq586" podStartSLOduration=4.1735496229999995 podStartE2EDuration="12.3795588s" totalImagesPullingTime="8.206009177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:00.406598088 +0000 UTC m=+11.841913818" watchObservedRunningTime="2026-10-30 21:42:01.3795588 +0000 UTC m=+12.814874529" Oct 30 21:42:02.178616 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:02.178530 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" event={"ID":"df4f0be9-a14d-4120-a10f-c815830aef30","Type":"ContainerStarted","Data":"4d51202f1c6817cdc4819db6afa1e2afe8d7abdf615996cde2c61510f6eca870"} Oct 30 21:42:02.180469 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:02.180435 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59qkx" event={"ID":"cde9eb3a-08fa-49c1-bd82-38b414b0235f","Type":"ContainerStarted","Data":"4ad2c52e85cfb4402733cc9d82038f9c76e6d25c66b8531416c862a8bc20e1b1"} Oct 30 21:42:02.297029 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:02.296982 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" podStartSLOduration=3.773438893 podStartE2EDuration="13.296968505s" totalImagesPullingTime="9.523529612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:02.296194259 +0000 UTC m=+13.731509989" watchObservedRunningTime="2026-10-30 21:42:02.296968505 +0000 UTC m=+13.732284271" Oct 30 21:42:02.411254 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:02.411204 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-59qkx" podStartSLOduration=4.752664186 podStartE2EDuration="13.411195093s" totalImagesPullingTime="8.658530907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:02.410880952 +0000 UTC m=+13.846196679" watchObservedRunningTime="2026-10-30 21:42:02.411195093 +0000 UTC m=+13.846510821" Oct 30 21:42:03.041100 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:03.041037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:03.041227 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:03.041132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:03.041294 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:03.041237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:03.041294 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:03.041255 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:03.041375 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:03.041338 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:03.041409 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:03.041370 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:03.185210 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:03.185181 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="2f8be2f8f17d12c25b4ba381028e089dba66c6c2f47cdaafed7f8920c13ae242" exitCode=0 Oct 30 21:42:03.185894 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:03.185265 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerDied","Data":"2f8be2f8f17d12c25b4ba381028e089dba66c6c2f47cdaafed7f8920c13ae242"} Oct 30 21:42:03.187087 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:03.187051 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ct72s" event={"ID":"7b5b5c38-04af-42ba-9845-41520e647cbe","Type":"ContainerStarted","Data":"02998e109501238f073328ecad6cc174bf356427a1fd23012c6e6e96da49c176"} Oct 30 21:42:03.426216 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:03.426121 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ct72s" podStartSLOduration=3.618635649 podStartE2EDuration="14.426106109s" totalImagesPullingTime="10.80747046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:03.425814244 +0000 UTC m=+14.861129985" watchObservedRunningTime="2026-10-30 21:42:03.426106109 +0000 UTC m=+14.861421835" Oct 30 21:42:05.038368 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:05.038338 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:05.038368 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:05.038368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:05.039123 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:05.038341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:05.039123 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:05.038464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:05.039123 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:05.038578 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:05.039123 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:05.038672 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:05.195755 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:05.195724 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="11f006004e19cf2be95d68b8aeba1ec13a9b55e1b2edfa7a1c045b71290f1a75" exitCode=0 Oct 30 21:42:05.195902 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:05.195780 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerDied","Data":"11f006004e19cf2be95d68b8aeba1ec13a9b55e1b2edfa7a1c045b71290f1a75"} Oct 30 21:42:06.806715 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:06.806562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:06.807235 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:06.806762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:06.807235 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.806642 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:06.807235 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.806889 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.806872306 +0000 UTC m=+34.242188026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:42:06.807235 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.806847 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:06.807235 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.806918 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret podName:5214b527-3150-40e4-9162-f712d602c83a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.806910846 +0000 UTC m=+34.242226564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret") pod "global-pull-secret-syncer-qdn6l" (UID: "5214b527-3150-40e4-9162-f712d602c83a") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:42:06.907298 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:06.907275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:06.907404 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.907393 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:42:06.907443 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.907409 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:42:06.907443 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.907418 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8v7rp for pod openshift-network-diagnostics/network-check-target-sz4m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:06.907577 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:06.907457 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp podName:55e5bc9a-1f53-4a57-88a5-446df5df92f0 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.907445282 +0000 UTC m=+34.342760991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8v7rp" (UniqueName: "kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp") pod "network-check-target-sz4m5" (UID: "55e5bc9a-1f53-4a57-88a5-446df5df92f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:42:07.040585 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.038714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:07.040585 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.038744 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:07.040585 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:07.038999 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:07.040585 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:07.039064 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:07.040585 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.038809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:07.040585 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:07.039139 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:07.203943 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.203898 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"e87cdde46606bb31e28d121f45eb7f02aaf0cc068573bef600db530008ed5923"} Oct 30 21:42:07.204096 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.203986 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"3b3589740802ce026bddaaa31065b5a2e6b8a56fdc7d4830c7ec2a422a2bc2aa"} Oct 30 21:42:07.204096 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.204038 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"ccc363415a7b32b93e16593f2ec25ee81be6d223b4d4d65ed1433d4d3f97a7a0"} Oct 30 21:42:07.204096 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.204083 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"68d02337687f84f07c9efabe8c18f5ef870ed743dbb2090fde7fa244ef00c753"} Oct 30 21:42:07.204204 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.204127 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"b7145b61bdfdfc6ddaba1ee862029d2decf454c52f7d4512ed71b8180017802f"} Oct 30 21:42:07.204204 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:07.204175 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"9cecfb6b8c778de546d2b74cbe4c2853e8815157403ad82e8b476c653f234a5d"} Oct 30 21:42:08.209035 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:08.208996 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="1cc1383b9c52122ef1f7293fc98487b2ab0565491ac13ebf07354654aa0bef25" exitCode=0 Oct 30 21:42:08.209454 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:08.209054 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerDied","Data":"1cc1383b9c52122ef1f7293fc98487b2ab0565491ac13ebf07354654aa0bef25"} Oct 30 21:42:09.039709 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:09.039684 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:09.039832 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:09.039752 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:09.039832 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:09.039796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:09.039956 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:09.039934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:09.039988 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:09.039965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:09.040038 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:09.040025 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:09.214015 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:09.213987 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"fe75fef4c5b1b41b3f41ead03fd1958d93dbbe34229d3c3cae84a50c76235ee9"} Oct 30 21:42:09.217056 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:09.217033 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="4de00840e426ff20a510f6bcbbeed7c436283b3635fb85c7159ec2cd0e0b169c" exitCode=0 Oct 30 21:42:09.217143 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:09.217079 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerDied","Data":"4de00840e426ff20a510f6bcbbeed7c436283b3635fb85c7159ec2cd0e0b169c"} Oct 30 21:42:10.222279 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:10.222248 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jw58x" event={"ID":"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7","Type":"ContainerStarted","Data":"e9074f040552adde55e466d3d2ebd289ccf277b59eea544038c7b234e297d2a8"} Oct 30 21:42:10.391932 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:10.391887 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jw58x" podStartSLOduration=9.045488835 podStartE2EDuration="21.391875261s" totalImagesPullingTime="12.346386426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:10.391344485 +0000 UTC m=+21.826660213" watchObservedRunningTime="2026-10-30 21:42:10.391875261 +0000 UTC m=+21.827190988" Oct 30 21:42:11.038189 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:11.038167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:11.038300 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:11.038197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:11.038300 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:11.038254 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:11.038300 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:11.038291 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:11.038405 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:11.038366 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:11.038505 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:11.038463 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:12.228792 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:12.228751 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" event={"ID":"fdb00f28-52e5-48e2-a034-7193a74f8157","Type":"ContainerStarted","Data":"dc0aee3fa1a18f1f6586306af0e10e319ae5e59e388f4d98c9e0a611387d66c0"} Oct 30 21:42:12.229626 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:12.228934 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:12.229626 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:12.228996 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:12.229626 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:12.229050 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:12.244390 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:12.244364 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:12.244522 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:12.244464 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:12.634990 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:12.634803 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" podStartSLOduration=8.313882337999999 podStartE2EDuration="23.634788621s" totalImagesPullingTime="15.320906283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.502712491 +0000 UTC m=+23.938028220" watchObservedRunningTime="2026-10-30 21:42:12.634788621 +0000 UTC m=+24.070104348" Oct 30 21:42:13.039190 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.039162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:13.039330 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.039196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:13.039330 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.039170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:13.039330 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:13.039280 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:13.039436 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:13.039335 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:13.039436 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:13.039388 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:13.302527 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.302447 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:42:13.302968 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.302589 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:13.303036 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.303024 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:42:13.881876 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.881844 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sz4m5"] Oct 30 21:42:13.882061 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.881950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:13.882141 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:13.882052 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:13.882628 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.882606 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g7hlg"] Oct 30 21:42:13.882715 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.882678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:13.882762 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:13.882748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:13.883066 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.883051 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdn6l"] Oct 30 21:42:13.883132 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:13.883122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:13.883212 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:13.883197 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:15.038841 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:15.038813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:15.039266 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:15.038817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:15.039266 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:15.038910 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:15.039406 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:15.039384 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:16.038927 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:16.038890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:16.039252 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:16.038992 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:17.038338 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:17.038310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:17.038338 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:17.038332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:17.038566 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:17.038426 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:17.038566 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:17.038557 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:18.039011 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:18.038989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:18.039390 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:18.039087 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sz4m5" podUID="55e5bc9a-1f53-4a57-88a5-446df5df92f0" Oct 30 21:42:19.038968 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.038930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:19.039210 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.039042 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g7hlg" podUID="c0a1d58f-d414-42c1-8505-79acb178f8cf" Oct 30 21:42:19.039210 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.039126 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:19.039669 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.039239 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qdn6l" podUID="5214b527-3150-40e4-9162-f712d602c83a" Oct 30 21:42:19.354508 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.354445 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeReady" Oct 30 21:42:19.354599 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.354561 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:42:19.467811 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.467514 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dj4nw"] Oct 30 21:42:19.470589 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.470574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" Oct 30 21:42:19.517721 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.517676 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tvk4h"] Oct 30 21:42:19.524667 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.524650 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ptmmz"] Oct 30 21:42:19.524776 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.524762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" Oct 30 21:42:19.527366 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.527348 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f"] Oct 30 21:42:19.527503 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.527472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" Oct 30 21:42:19.529832 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.529817 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7444c59677-nrb6q"] Oct 30 21:42:19.529954 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.529939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" Oct 30 21:42:19.532464 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.532447 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn"] Oct 30 21:42:19.532605 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.532589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.535237 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.535220 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7"] Oct 30 21:42:19.535339 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.535323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.537943 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.537928 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk"] Oct 30 21:42:19.538054 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.538039 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.547660 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.547645 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64975494cd-d2ggk"] Oct 30 21:42:19.547762 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.547749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.550302 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.550285 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-q69rp"] Oct 30 21:42:19.550408 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.550340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.553395 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.553374 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f"] Oct 30 21:42:19.555078 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.555044 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.556413 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.556391 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" Oct 30 21:42:19.566476 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.566436 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.566761 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.566742 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.568526 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.568444 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5xpsq"] Oct 30 21:42:19.571448 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.571428 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675b86c7dc-q72nj"] Oct 30 21:42:19.573124 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.571578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:19.574973 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.574958 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn"] Oct 30 21:42:19.575089 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.575070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.578993 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.578960 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zxfcs"] Oct 30 21:42:19.579241 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.579227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.582796 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.582775 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gmcqj"] Oct 30 21:42:19.582902 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.582884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.585845 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.585830 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pb7j7"] Oct 30 21:42:19.587218 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.587200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:19.588590 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588573 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tvk4h"] Oct 30 21:42:19.588672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588600 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dj4nw"] Oct 30 21:42:19.588672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588617 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ptmmz"] Oct 30 21:42:19.588672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588633 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f"] Oct 30 21:42:19.588672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588645 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn"] Oct 30 21:42:19.588672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588657 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk"] Oct 30 21:42:19.588672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.588672 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.588673 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7444c59677-nrb6q"] Oct 30 21:42:19.591670 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.591656 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:42:19.591817 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.591802 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:42:19.593581 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.593566 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64975494cd-d2ggk"] Oct 30 21:42:19.603236 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-serving-cert\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.603305 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-config\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.603305 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5qk\" (UniqueName: \"kubernetes.io/projected/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-kube-api-access-4r5qk\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.603305 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhmd7\" (UniqueName: \"kubernetes.io/projected/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9-kube-api-access-lhmd7\") pod \"kas-connection-checker-685895c7c9-ptmmz\" (UID: \"f8ea0e11-f736-4db0-9ffe-d46ed7479dd9\") " pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" Oct 30 21:42:19.603390 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.603390 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf557\" (UniqueName: \"kubernetes.io/projected/7dd70011-6085-49a8-b665-cb33157874fe-kube-api-access-pf557\") pod \"kas-connection-checker-685895c7c9-dj4nw\" (UID: \"7dd70011-6085-49a8-b665-cb33157874fe\") " pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" Oct 30 21:42:19.603533 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhq24\" (UniqueName: \"kubernetes.io/projected/3b72356e-8b7b-4392-859e-5ac980f759ef-kube-api-access-xhq24\") pod \"volume-data-source-validator-58c9f5f46d-85c8f\" (UID: \"3b72356e-8b7b-4392-859e-5ac980f759ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" Oct 30 21:42:19.603575 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7mng\" (UniqueName: \"kubernetes.io/projected/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-kube-api-access-z7mng\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.603650 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-stats-auth\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.603683 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.603683 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.603769 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p6rh\" (UniqueName: \"kubernetes.io/projected/3be119ed-6228-46e8-a71b-8f7fb6e812f8-kube-api-access-7p6rh\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.603802 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjnd\" (UniqueName: \"kubernetes.io/projected/b10b2b7a-ecaf-434a-8248-0bdfd7df1667-kube-api-access-zrjnd\") pod \"kas-connection-checker-685895c7c9-tvk4h\" (UID: \"b10b2b7a-ecaf-434a-8248-0bdfd7df1667\") " pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" Oct 30 21:42:19.603832 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.603863 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.603842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-default-certificate\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.615362 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.615342 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zs9h6\"" Oct 30 21:42:19.615362 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.615342 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rpk5c\"" Oct 30 21:42:19.620105 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.620086 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:42:19.620371 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.620315 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:42:19.620859 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.620675 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xcxlp\"" Oct 30 21:42:19.620859 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.620724 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bz8qf\"" Oct 30 21:42:19.620859 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.620741 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hln5f\"" Oct 30 21:42:19.620859 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.620742 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lbgb5\"" Oct 30 21:42:19.621095 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621080 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:42:19.621223 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621142 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:42:19.621223 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621167 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:42:19.621369 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621342 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:42:19.621476 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621461 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:42:19.621596 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621523 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:42:19.621596 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621550 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tzxpl\"" Oct 30 21:42:19.621694 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.621599 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6xflp\"" Oct 30 21:42:19.626407 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.626373 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.626407 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.626383 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:42:19.626699 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.626668 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.626782 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.626733 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:42:19.626782 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.626746 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.626948 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.626930 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.629722 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.629701 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:42:19.629798 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.629745 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:42:19.630016 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.629990 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.630484 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.630263 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.630484 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.630010 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.630628 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.630531 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.632593 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.630832 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.632593 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.630913 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:42:19.632593 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.631103 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.632593 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.631254 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.632593 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.631322 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:42:19.632967 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.632946 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.638262 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.638244 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.669909 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.669884 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.670627 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.670613 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:42:19.673336 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.673319 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fn8lc\"" Oct 30 21:42:19.676131 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.676109 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:42:19.679908 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.679893 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:42:19.680682 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.680666 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmcqj"] Oct 30 21:42:19.683817 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.683796 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7"] Oct 30 21:42:19.684510 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.684473 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675b86c7dc-q72nj"] Oct 30 21:42:19.685317 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.685296 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pb7j7"] Oct 30 21:42:19.686326 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.686307 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-q69rp"] Oct 30 21:42:19.687027 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.687010 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn"] Oct 30 21:42:19.691442 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.691422 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f"] Oct 30 21:42:19.691755 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.691733 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:42:19.691836 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.691782 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tpd8h\"" Oct 30 21:42:19.692014 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.691998 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6j5v7\"" Oct 30 21:42:19.692014 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.692011 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cmjqq\"" Oct 30 21:42:19.692126 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.692005 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:42:19.692126 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.692015 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:42:19.692126 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.692043 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9lfqr\"" Oct 30 21:42:19.692126 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.692055 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:42:19.692360 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.692203 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5xpsq"] Oct 30 21:42:19.701250 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.701228 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zxfcs"] Oct 30 21:42:19.704646 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmd7\" (UniqueName: \"kubernetes.io/projected/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9-kube-api-access-lhmd7\") pod \"kas-connection-checker-685895c7c9-ptmmz\" (UID: \"f8ea0e11-f736-4db0-9ffe-d46ed7479dd9\") " pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" Oct 30 21:42:19.704745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf557\" (UniqueName: \"kubernetes.io/projected/7dd70011-6085-49a8-b665-cb33157874fe-kube-api-access-pf557\") pod \"kas-connection-checker-685895c7c9-dj4nw\" (UID: \"7dd70011-6085-49a8-b665-cb33157874fe\") " pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" Oct 30 21:42:19.704745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.704745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.704745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvl8j\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.704745 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.704982 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhq24\" (UniqueName: \"kubernetes.io/projected/3b72356e-8b7b-4392-859e-5ac980f759ef-kube-api-access-xhq24\") pod \"volume-data-source-validator-58c9f5f46d-85c8f\" (UID: \"3b72356e-8b7b-4392-859e-5ac980f759ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" Oct 30 21:42:19.704982 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mng\" (UniqueName: \"kubernetes.io/projected/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-kube-api-access-z7mng\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.705068 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.704999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dtl\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.705068 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gjp\" (UniqueName: \"kubernetes.io/projected/d0ee0cf5-2de4-493c-85fe-f97c527d5635-kube-api-access-54gjp\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.705147 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-stats-auth\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.705180 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.705180 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.705248 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:19.705248 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81526e36-032f-4f22-a94e-5e48680b7bc7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:19.705308 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.705280 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:19.705339 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.705332 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.205313809 +0000 UTC m=+31.640629530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:19.705396 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-tmp\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.705460 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.705513 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/061b0a6d-64e8-4a45-8ada-59d74f6023a2-cert\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:19.705566 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.705604 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.705641 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-default-certificate\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.705683 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkg7\" (UniqueName: \"kubernetes.io/projected/8b8dc362-93da-466a-b4bd-7d33a25a00e4-kube-api-access-xmkg7\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.705683 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.705806 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.705848 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-service-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.705848 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.705912 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsvd5\" (UniqueName: \"kubernetes.io/projected/95ce4f09-438e-4534-aeae-d427c599641a-kube-api-access-zsvd5\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.705967 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.705922 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.205908456 +0000 UTC m=+31.641224183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:19.706025 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5qk\" (UniqueName: \"kubernetes.io/projected/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-kube-api-access-4r5qk\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.706025 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.705992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-snapshots\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.706025 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.706116 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrzf\" (UniqueName: \"kubernetes.io/projected/c9a317df-0311-4ba6-bbb1-7556663b7bb5-kube-api-access-dvrzf\") pod \"network-check-source-6f78d59f99-7wr6f\" (UID: \"c9a317df-0311-4ba6-bbb1-7556663b7bb5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" Oct 30 21:42:19.706116 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.706185 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0ee0cf5-2de4-493c-85fe-f97c527d5635-serving-cert\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.706185 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-trusted-ca\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.706267 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.706267 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.706353 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-config\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.706353 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.706429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p6rh\" (UniqueName: \"kubernetes.io/projected/3be119ed-6228-46e8-a71b-8f7fb6e812f8-kube-api-access-7p6rh\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.706429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ce4f09-438e-4534-aeae-d427c599641a-config-volume\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.706429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.706562 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ce4f09-438e-4534-aeae-d427c599641a-tmp-dir\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.706562 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjnd\" (UniqueName: \"kubernetes.io/projected/b10b2b7a-ecaf-434a-8248-0bdfd7df1667-kube-api-access-zrjnd\") pod \"kas-connection-checker-685895c7c9-tvk4h\" (UID: \"b10b2b7a-ecaf-434a-8248-0bdfd7df1667\") " pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" Oct 30 21:42:19.706627 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.706663 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.706717 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.706769 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kkzt\" (UniqueName: \"kubernetes.io/projected/061b0a6d-64e8-4a45-8ada-59d74f6023a2-kube-api-access-8kkzt\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:19.706769 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.706865 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f607097-ece8-4833-8f34-361ac77ca2ee-serving-cert\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.706865 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs4q\" (UniqueName: \"kubernetes.io/projected/1f607097-ece8-4833-8f34-361ac77ca2ee-kube-api-access-vcs4q\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.706865 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8dc362-93da-466a-b4bd-7d33a25a00e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.706985 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g22b\" (UniqueName: \"kubernetes.io/projected/ef109382-58f3-45a9-bbd6-ef811dafc3d1-kube-api-access-5g22b\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.706985 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.706985 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-serving-cert\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.706985 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.706981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-config\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.707166 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.707017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.707166 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.707046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.707782 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.707716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-config\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.709263 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.709241 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.709369 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.709274 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-serving-cert\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.709369 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.709357 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-default-certificate\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.709464 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.709397 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-stats-auth\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.723594 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.723573 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.723594 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.723573 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.723738 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.723599 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:42:19.723877 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.723864 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:42:19.723942 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.723894 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:42:19.723942 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.723897 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.724061 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.724045 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.724113 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.724067 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:42:19.724245 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.724231 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:42:19.725451 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.725432 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:42:19.762964 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.762844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmd7\" (UniqueName: \"kubernetes.io/projected/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9-kube-api-access-lhmd7\") pod \"kas-connection-checker-685895c7c9-ptmmz\" (UID: \"f8ea0e11-f736-4db0-9ffe-d46ed7479dd9\") " pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" Oct 30 21:42:19.764250 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.764231 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5qk\" (UniqueName: \"kubernetes.io/projected/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-kube-api-access-4r5qk\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.764610 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.764591 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf557\" (UniqueName: \"kubernetes.io/projected/7dd70011-6085-49a8-b665-cb33157874fe-kube-api-access-pf557\") pod \"kas-connection-checker-685895c7c9-dj4nw\" (UID: \"7dd70011-6085-49a8-b665-cb33157874fe\") " pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" Oct 30 21:42:19.764667 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.764655 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mng\" (UniqueName: \"kubernetes.io/projected/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-kube-api-access-z7mng\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.765070 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.765052 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjnd\" (UniqueName: \"kubernetes.io/projected/b10b2b7a-ecaf-434a-8248-0bdfd7df1667-kube-api-access-zrjnd\") pod \"kas-connection-checker-685895c7c9-tvk4h\" (UID: \"b10b2b7a-ecaf-434a-8248-0bdfd7df1667\") " pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" Oct 30 21:42:19.765137 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.765122 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p6rh\" (UniqueName: \"kubernetes.io/projected/3be119ed-6228-46e8-a71b-8f7fb6e812f8-kube-api-access-7p6rh\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:19.771241 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.771222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhq24\" (UniqueName: \"kubernetes.io/projected/3b72356e-8b7b-4392-859e-5ac980f759ef-kube-api-access-xhq24\") pod \"volume-data-source-validator-58c9f5f46d-85c8f\" (UID: \"3b72356e-8b7b-4392-859e-5ac980f759ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" Oct 30 21:42:19.782095 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.782078 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" Oct 30 21:42:19.807762 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.807741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.807873 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.807774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0ee0cf5-2de4-493c-85fe-f97c527d5635-serving-cert\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.807873 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.807800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-trusted-ca\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.807873 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.807830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.807873 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.807856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.808095 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-config\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.808156 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.808156 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ce4f09-438e-4534-aeae-d427c599641a-config-volume\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.808257 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.808257 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ce4f09-438e-4534-aeae-d427c599641a-tmp-dir\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.808257 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.808424 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.808424 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.808424 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kkzt\" (UniqueName: \"kubernetes.io/projected/061b0a6d-64e8-4a45-8ada-59d74f6023a2-kube-api-access-8kkzt\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:19.808424 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f607097-ece8-4833-8f34-361ac77ca2ee-serving-cert\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.808424 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs4q\" (UniqueName: \"kubernetes.io/projected/1f607097-ece8-4833-8f34-361ac77ca2ee-kube-api-access-vcs4q\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.808424 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8dc362-93da-466a-b4bd-7d33a25a00e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g22b\" (UniqueName: \"kubernetes.io/projected/ef109382-58f3-45a9-bbd6-ef811dafc3d1-kube-api-access-5g22b\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvl8j\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dtl\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.808712 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gjp\" (UniqueName: \"kubernetes.io/projected/d0ee0cf5-2de4-493c-85fe-f97c527d5635-kube-api-access-54gjp\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808721 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81526e36-032f-4f22-a94e-5e48680b7bc7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-tmp\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/061b0a6d-64e8-4a45-8ada-59d74f6023a2-cert\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-trusted-ca\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.808962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.809183 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.809127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ce4f09-438e-4534-aeae-d427c599641a-config-volume\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.809241 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.809257 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.809281 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.809280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.809308 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.309289743 +0000 UTC m=+31.744605468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.809370 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.809384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.809408 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.309396321 +0000 UTC m=+31.744712047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:19.809641 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.809473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.809898 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.809837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.809929 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.809900 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:19.809959 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.809936 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.309925926 +0000 UTC m=+31.745241644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:19.809996 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.809949 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.810025 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810011 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81526e36-032f-4f22-a94e-5e48680b7bc7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:19.810055 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.810045 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:19.810109 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.810090 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.310075145 +0000 UTC m=+31.745390854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:19.810182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkg7\" (UniqueName: \"kubernetes.io/projected/8b8dc362-93da-466a-b4bd-7d33a25a00e4-kube-api-access-xmkg7\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.810182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.810281 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810196 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-service-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.810281 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvd5\" (UniqueName: \"kubernetes.io/projected/95ce4f09-438e-4534-aeae-d427c599641a-kube-api-access-zsvd5\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.810281 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-tmp\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.810281 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-snapshots\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.810528 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.810528 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810311 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ce4f09-438e-4534-aeae-d427c599641a-tmp-dir\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.810528 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrzf\" (UniqueName: \"kubernetes.io/projected/c9a317df-0311-4ba6-bbb1-7556663b7bb5-kube-api-access-dvrzf\") pod \"network-check-source-6f78d59f99-7wr6f\" (UID: \"c9a317df-0311-4ba6-bbb1-7556663b7bb5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" Oct 30 21:42:19.810528 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.810442 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:19.810528 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.810452 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:19.810528 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.810482 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.31047263 +0000 UTC m=+31.745788336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:19.810954 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-service-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.811009 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.810972 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:19.811060 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.810988 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-snapshots\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.811060 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:19.811017 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.311003414 +0000 UTC m=+31.746319122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:19.811174 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.811149 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.811984 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.811567 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-config\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.811984 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.811934 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8dc362-93da-466a-b4bd-7d33a25a00e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.813014 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.812995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.813533 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.813510 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0ee0cf5-2de4-493c-85fe-f97c527d5635-serving-cert\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.814079 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.813824 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f607097-ece8-4833-8f34-361ac77ca2ee-serving-cert\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.814079 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.814030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.814079 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.814032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.814425 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.814400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.815198 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.815181 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/061b0a6d-64e8-4a45-8ada-59d74f6023a2-cert\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:19.837005 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.836988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" Oct 30 21:42:19.842846 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.842823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" Oct 30 21:42:19.848975 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.848958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" Oct 30 21:42:19.865525 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.865483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:19.867095 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.867072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.869182 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.869036 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dtl\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.869824 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.869766 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gjp\" (UniqueName: \"kubernetes.io/projected/d0ee0cf5-2de4-493c-85fe-f97c527d5635-kube-api-access-54gjp\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.871009 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.870926 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g22b\" (UniqueName: \"kubernetes.io/projected/ef109382-58f3-45a9-bbd6-ef811dafc3d1-kube-api-access-5g22b\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:19.871857 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.871832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kkzt\" (UniqueName: \"kubernetes.io/projected/061b0a6d-64e8-4a45-8ada-59d74f6023a2-kube-api-access-8kkzt\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:19.872380 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.872333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvd5\" (UniqueName: \"kubernetes.io/projected/95ce4f09-438e-4534-aeae-d427c599641a-kube-api-access-zsvd5\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:19.873056 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.872981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:19.873056 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.873015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvl8j\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:19.873941 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.873810 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkg7\" (UniqueName: \"kubernetes.io/projected/8b8dc362-93da-466a-b4bd-7d33a25a00e4-kube-api-access-xmkg7\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:19.874109 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.874069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:19.874157 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.874112 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrzf\" (UniqueName: \"kubernetes.io/projected/c9a317df-0311-4ba6-bbb1-7556663b7bb5-kube-api-access-dvrzf\") pod \"network-check-source-6f78d59f99-7wr6f\" (UID: \"c9a317df-0311-4ba6-bbb1-7556663b7bb5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" Oct 30 21:42:19.880527 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.875787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs4q\" (UniqueName: \"kubernetes.io/projected/1f607097-ece8-4833-8f34-361ac77ca2ee-kube-api-access-vcs4q\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.897758 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.897500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:19.914901 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.908945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" Oct 30 21:42:19.927560 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.927536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:19.941102 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:19.940999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:20.040086 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.039085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:20.091250 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.091229 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-msl6j\"" Oct 30 21:42:20.169479 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.169457 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dj4nw"] Oct 30 21:42:20.172094 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.172074 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd70011_6085_49a8_b665_cb33157874fe.slice/crio-fd8c3848a20067531f25b3224285e016af80ec596ce8e37162b777b4999bf0ef WatchSource:0}: Error finding container fd8c3848a20067531f25b3224285e016af80ec596ce8e37162b777b4999bf0ef: Status 404 returned error can't find the container with id fd8c3848a20067531f25b3224285e016af80ec596ce8e37162b777b4999bf0ef Oct 30 21:42:20.217843 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.217814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:20.217987 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.217964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:20.217987 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.217980 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.217959489 +0000 UTC m=+32.653275209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:20.218123 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.218050 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:20.218123 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.218079 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.218071224 +0000 UTC m=+32.653386930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:20.220543 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.220521 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ptmmz"] Oct 30 21:42:20.223364 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.223342 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ea0e11_f736_4db0_9ffe_d46ed7479dd9.slice/crio-d6cc4b1ddd7a49d3c43e4ab52b41de21c708e509c7d62e95af84a80ceaed918b WatchSource:0}: Error finding container d6cc4b1ddd7a49d3c43e4ab52b41de21c708e509c7d62e95af84a80ceaed918b: Status 404 returned error can't find the container with id d6cc4b1ddd7a49d3c43e4ab52b41de21c708e509c7d62e95af84a80ceaed918b Oct 30 21:42:20.232193 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.232172 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-tvk4h"] Oct 30 21:42:20.236586 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.236563 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10b2b7a_ecaf_434a_8248_0bdfd7df1667.slice/crio-231a6f2bbe3c68df30cba0c1fd683ae23b63efe5abc26da7c72024fd81c236b2 WatchSource:0}: Error finding container 231a6f2bbe3c68df30cba0c1fd683ae23b63efe5abc26da7c72024fd81c236b2: Status 404 returned error can't find the container with id 231a6f2bbe3c68df30cba0c1fd683ae23b63efe5abc26da7c72024fd81c236b2 Oct 30 21:42:20.246716 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.246653 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" event={"ID":"f8ea0e11-f736-4db0-9ffe-d46ed7479dd9","Type":"ContainerStarted","Data":"d6cc4b1ddd7a49d3c43e4ab52b41de21c708e509c7d62e95af84a80ceaed918b"} Oct 30 21:42:20.248361 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.248311 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" event={"ID":"b10b2b7a-ecaf-434a-8248-0bdfd7df1667","Type":"ContainerStarted","Data":"231a6f2bbe3c68df30cba0c1fd683ae23b63efe5abc26da7c72024fd81c236b2"} Oct 30 21:42:20.249754 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.249701 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" event={"ID":"7dd70011-6085-49a8-b665-cb33157874fe","Type":"ContainerStarted","Data":"fd8c3848a20067531f25b3224285e016af80ec596ce8e37162b777b4999bf0ef"} Oct 30 21:42:20.277929 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.277655 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn"] Oct 30 21:42:20.281379 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.281355 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ea4697_1a09_47e3_b6bc_ed44dae88ded.slice/crio-90e6dae0754dbefceaef28d17c2308a64bf4f064a4636cecce6cfc764c7abf91 WatchSource:0}: Error finding container 90e6dae0754dbefceaef28d17c2308a64bf4f064a4636cecce6cfc764c7abf91: Status 404 returned error can't find the container with id 90e6dae0754dbefceaef28d17c2308a64bf4f064a4636cecce6cfc764c7abf91 Oct 30 21:42:20.292398 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.292372 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f"] Oct 30 21:42:20.293543 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.293519 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7"] Oct 30 21:42:20.297026 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.296999 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adf6cb3_60c7_471c_ba9e_62c236e4c7c4.slice/crio-82e57fc8e5eec381f62314957def4f0e1b865b7151c898fb9527dfb656494734 WatchSource:0}: Error finding container 82e57fc8e5eec381f62314957def4f0e1b865b7151c898fb9527dfb656494734: Status 404 returned error can't find the container with id 82e57fc8e5eec381f62314957def4f0e1b865b7151c898fb9527dfb656494734 Oct 30 21:42:20.298659 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.298636 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b72356e_8b7b_4392_859e_5ac980f759ef.slice/crio-8b29de7f55f2b2bc89c23f0d89097a922ff5e7fce245d28e3ecd1be3e58bf3cc WatchSource:0}: Error finding container 8b29de7f55f2b2bc89c23f0d89097a922ff5e7fce245d28e3ecd1be3e58bf3cc: Status 404 returned error can't find the container with id 8b29de7f55f2b2bc89c23f0d89097a922ff5e7fce245d28e3ecd1be3e58bf3cc Oct 30 21:42:20.318362 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.318341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:20.318442 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.318371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:20.318442 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.318392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:20.318576 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318507 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:20.318576 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318519 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:20.318576 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318526 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:20.318576 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318535 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:20.318576 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.318556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:20.318576 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318573 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.318553745 +0000 UTC m=+32.753869482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:20.318858 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318611 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.318592288 +0000 UTC m=+32.753907998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:20.318858 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318627 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:20.318858 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318636 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.318621595 +0000 UTC m=+32.753937316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:20.318858 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318665 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.318650777 +0000 UTC m=+32.753966498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:20.318858 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.318729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:20.318858 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.318776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:20.319096 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318868 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:20.319096 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318928 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.318914204 +0000 UTC m=+32.754229926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:20.319096 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318938 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:20.319096 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318950 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:20.319096 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:20.318983 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.318971307 +0000 UTC m=+32.754287033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:20.325501 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.325461 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-q69rp"] Oct 30 21:42:20.336081 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.336055 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ee0cf5_2de4_493c_85fe_f97c527d5635.slice/crio-f8dc369c7cb6c394f7c43a41b51a09160b673b8530af9fac486e234c92a09821 WatchSource:0}: Error finding container f8dc369c7cb6c394f7c43a41b51a09160b673b8530af9fac486e234c92a09821: Status 404 returned error can't find the container with id f8dc369c7cb6c394f7c43a41b51a09160b673b8530af9fac486e234c92a09821 Oct 30 21:42:20.358222 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.358201 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zxfcs"] Oct 30 21:42:20.362403 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.362379 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f607097_ece8_4833_8f34_361ac77ca2ee.slice/crio-35a10c0396b671cee440376eb603bb60139357665a2765130d33402f25d3d2a0 WatchSource:0}: Error finding container 35a10c0396b671cee440376eb603bb60139357665a2765130d33402f25d3d2a0: Status 404 returned error can't find the container with id 35a10c0396b671cee440376eb603bb60139357665a2765130d33402f25d3d2a0 Oct 30 21:42:20.364109 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.363950 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmcqj"] Oct 30 21:42:20.366514 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:20.366474 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f"] Oct 30 21:42:20.369621 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.369600 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a317df_0311_4ba6_bbb1_7556663b7bb5.slice/crio-f14f955699ec5b9843748aa7672231f820097cae6cdc98b75ace3e4b09bc6a23 WatchSource:0}: Error finding container f14f955699ec5b9843748aa7672231f820097cae6cdc98b75ace3e4b09bc6a23: Status 404 returned error can't find the container with id f14f955699ec5b9843748aa7672231f820097cae6cdc98b75ace3e4b09bc6a23 Oct 30 21:42:20.370039 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:20.370014 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061b0a6d_64e8_4a45_8ada_59d74f6023a2.slice/crio-5c84179f08afd6dea6fa37fbfe373e467b4d299a6efb1b4cc33c719688f76c86 WatchSource:0}: Error finding container 5c84179f08afd6dea6fa37fbfe373e467b4d299a6efb1b4cc33c719688f76c86: Status 404 returned error can't find the container with id 5c84179f08afd6dea6fa37fbfe373e467b4d299a6efb1b4cc33c719688f76c86 Oct 30 21:42:21.038541 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.038512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:21.038701 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.038683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:21.095159 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.091546 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:42:21.095159 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.091691 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcz6k\"" Oct 30 21:42:21.095159 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.091908 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-spllf\"" Oct 30 21:42:21.095159 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.092223 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:42:21.227190 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.227153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:21.227352 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.227252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:21.227425 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.227412 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.227391853 +0000 UTC m=+34.662707573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:21.227621 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.227547 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:21.227621 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.227590 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.227578092 +0000 UTC m=+34.662893813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:21.256580 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.255949 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" event={"ID":"f8ea0e11-f736-4db0-9ffe-d46ed7479dd9","Type":"ContainerStarted","Data":"2c0157fe18fe2d6c810b3736ed7e4ac3d88dad33a59a1c3134f10411fd02b261"} Oct 30 21:42:21.260481 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.260444 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" event={"ID":"c9a317df-0311-4ba6-bbb1-7556663b7bb5","Type":"ContainerStarted","Data":"f14f955699ec5b9843748aa7672231f820097cae6cdc98b75ace3e4b09bc6a23"} Oct 30 21:42:21.268936 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.268382 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" event={"ID":"b10b2b7a-ecaf-434a-8248-0bdfd7df1667","Type":"ContainerStarted","Data":"16b92c2595df53dfacf3453e9e6d2486e17117960487c0acaac09d7da50fafbe"} Oct 30 21:42:21.272827 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.272800 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" event={"ID":"7dd70011-6085-49a8-b665-cb33157874fe","Type":"ContainerStarted","Data":"0c1cea2d5432a4ec353bd01a86c60c4c1bc4876c80c58226a3889a517f9547b3"} Oct 30 21:42:21.275099 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.275037 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmcqj" event={"ID":"061b0a6d-64e8-4a45-8ada-59d74f6023a2","Type":"ContainerStarted","Data":"5c84179f08afd6dea6fa37fbfe373e467b4d299a6efb1b4cc33c719688f76c86"} Oct 30 21:42:21.287014 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.286990 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zxfcs" event={"ID":"1f607097-ece8-4833-8f34-361ac77ca2ee","Type":"ContainerStarted","Data":"35a10c0396b671cee440376eb603bb60139357665a2765130d33402f25d3d2a0"} Oct 30 21:42:21.288736 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.288676 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" event={"ID":"d0ee0cf5-2de4-493c-85fe-f97c527d5635","Type":"ContainerStarted","Data":"f8dc369c7cb6c394f7c43a41b51a09160b673b8530af9fac486e234c92a09821"} Oct 30 21:42:21.291778 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.291754 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" event={"ID":"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4","Type":"ContainerStarted","Data":"82e57fc8e5eec381f62314957def4f0e1b865b7151c898fb9527dfb656494734"} Oct 30 21:42:21.294820 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.294790 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" event={"ID":"f9ea4697-1a09-47e3-b6bc-ed44dae88ded","Type":"ContainerStarted","Data":"90e6dae0754dbefceaef28d17c2308a64bf4f064a4636cecce6cfc764c7abf91"} Oct 30 21:42:21.297222 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.297143 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" event={"ID":"3b72356e-8b7b-4392-859e-5ac980f759ef","Type":"ContainerStarted","Data":"8b29de7f55f2b2bc89c23f0d89097a922ff5e7fce245d28e3ecd1be3e58bf3cc"} Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.328504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.328549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.328600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.328634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.328701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.328742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.328939 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.328955 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329007 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.328989715 +0000 UTC m=+34.764305428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329075 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329084 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329116 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.329103385 +0000 UTC m=+34.764419096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329174 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329201 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.329191677 +0000 UTC m=+34.764507387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329936 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.329984 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.32996988 +0000 UTC m=+34.765285588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:21.330202 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.330048 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:21.331074 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.330080 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.330068968 +0000 UTC m=+34.765384679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:21.331074 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.330137 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:21.331074 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:21.330167 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:23.330157433 +0000 UTC m=+34.765473140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:21.674771 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.674676 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" podStartSLOduration=385.674657645 podStartE2EDuration="6m25.674657645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:21.674183096 +0000 UTC m=+33.109498825" watchObservedRunningTime="2026-10-30 21:42:21.674657645 +0000 UTC m=+33.109973375" Oct 30 21:42:21.795277 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.795221 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" podStartSLOduration=385.795203545 podStartE2EDuration="6m25.795203545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:21.794799563 +0000 UTC m=+33.230115293" watchObservedRunningTime="2026-10-30 21:42:21.795203545 +0000 UTC m=+33.230519274" Oct 30 21:42:21.917820 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:21.916765 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" podStartSLOduration=385.916744915 podStartE2EDuration="6m25.916744915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:21.915728494 +0000 UTC m=+33.351044223" watchObservedRunningTime="2026-10-30 21:42:21.916744915 +0000 UTC m=+33.352060645" Oct 30 21:42:22.848753 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:22.847818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:22.848753 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:22.847940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:22.848753 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:22.848082 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:22.848753 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:22.848143 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:42:54.848128304 +0000 UTC m=+66.283444027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : secret "metrics-daemon-secret" not found Oct 30 21:42:22.851609 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:22.851552 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:22.856770 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:22.856571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:22.949325 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:22.948694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:22.953616 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:22.953561 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:23.074773 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.074505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:23.252205 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.251886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:23.252205 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.252036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:23.252205 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.252079 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.252054151 +0000 UTC m=+38.687369867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:23.252205 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.252134 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:23.252205 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.252181 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.252167073 +0000 UTC m=+38.687482779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:23.304607 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.304577 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qdn6l"] Oct 30 21:42:23.312882 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:23.312818 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5214b527_3150_40e4_9162_f712d602c83a.slice/crio-73f6307ec4bbb6b25369f79d30de80a63f9ac59d5da2dfc8638800adc1b005c0 WatchSource:0}: Error finding container 73f6307ec4bbb6b25369f79d30de80a63f9ac59d5da2dfc8638800adc1b005c0: Status 404 returned error can't find the container with id 73f6307ec4bbb6b25369f79d30de80a63f9ac59d5da2dfc8638800adc1b005c0 Oct 30 21:42:23.316196 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.316090 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" event={"ID":"3b72356e-8b7b-4392-859e-5ac980f759ef","Type":"ContainerStarted","Data":"2101656c6af33939af7d5316b658fe104890f3c3d2965994383b691b9b9c5ec8"} Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.352482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.352547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.352582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.352617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.352668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.352710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.352893 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.352908 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.352961 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.352942194 +0000 UTC m=+38.788257913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353021 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353032 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353064 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.353052717 +0000 UTC m=+38.788368437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353123 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353153 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.353143589 +0000 UTC m=+38.788459309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353206 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353235 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.353225929 +0000 UTC m=+38.788541674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:23.353429 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353287 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:23.354359 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353315 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.353305693 +0000 UTC m=+38.788621735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:23.354359 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353364 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:23.354359 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:23.353399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.353389256 +0000 UTC m=+38.788704976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:23.435271 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.435170 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" podStartSLOduration=331.247676164 podStartE2EDuration="5m33.435151017s" totalImagesPullingTime="2.187474853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:23.432905819 +0000 UTC m=+34.868221547" watchObservedRunningTime="2026-10-30 21:42:23.435151017 +0000 UTC m=+34.870466746" Oct 30 21:42:23.520688 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:23.520661 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sz4m5"] Oct 30 21:42:23.525151 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:23.525120 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e5bc9a_1f53_4a57_88a5_446df5df92f0.slice/crio-8abe5c46b8139fd38061759341005be3acf9fac5347f1d30e7caf27a4580fb42 WatchSource:0}: Error finding container 8abe5c46b8139fd38061759341005be3acf9fac5347f1d30e7caf27a4580fb42: Status 404 returned error can't find the container with id 8abe5c46b8139fd38061759341005be3acf9fac5347f1d30e7caf27a4580fb42 Oct 30 21:42:24.323840 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:24.323801 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qdn6l" event={"ID":"5214b527-3150-40e4-9162-f712d602c83a","Type":"ContainerStarted","Data":"73f6307ec4bbb6b25369f79d30de80a63f9ac59d5da2dfc8638800adc1b005c0"} Oct 30 21:42:24.327600 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:24.327525 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sz4m5" event={"ID":"55e5bc9a-1f53-4a57-88a5-446df5df92f0","Type":"ContainerStarted","Data":"8abe5c46b8139fd38061759341005be3acf9fac5347f1d30e7caf27a4580fb42"} Oct 30 21:42:25.332479 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:25.332449 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/0.log" Oct 30 21:42:25.332888 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:25.332518 2579 generic.go:356] "Generic (PLEG): container finished" podID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" containerID="776af6e1d06fca5704fb1ad316ab71a6b907f18fa099353dd14c5f266a71e085" exitCode=255 Oct 30 21:42:25.332888 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:25.332607 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" event={"ID":"d0ee0cf5-2de4-493c-85fe-f97c527d5635","Type":"ContainerDied","Data":"776af6e1d06fca5704fb1ad316ab71a6b907f18fa099353dd14c5f266a71e085"} Oct 30 21:42:25.332888 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:25.332789 2579 scope.go:122] "RemoveContainer" containerID="776af6e1d06fca5704fb1ad316ab71a6b907f18fa099353dd14c5f266a71e085" Oct 30 21:42:26.340531 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.340480 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmcqj" event={"ID":"061b0a6d-64e8-4a45-8ada-59d74f6023a2","Type":"ContainerStarted","Data":"786de77d9bcd9e93bcebe40ac59219456fc03e7e179f5ca37f357d04595594b9"} Oct 30 21:42:26.343555 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.343522 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zxfcs" event={"ID":"1f607097-ece8-4833-8f34-361ac77ca2ee","Type":"ContainerStarted","Data":"a1a0cf21b42f4cf0d631f4d7adc864591f42fd43200d33ebfb203d0c911d9704"} Oct 30 21:42:26.346558 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.346532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/1.log" Oct 30 21:42:26.347267 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.347245 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/0.log" Oct 30 21:42:26.347346 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.347288 2579 generic.go:356] "Generic (PLEG): container finished" podID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" containerID="a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4" exitCode=255 Oct 30 21:42:26.347398 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.347369 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" event={"ID":"d0ee0cf5-2de4-493c-85fe-f97c527d5635","Type":"ContainerDied","Data":"a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4"} Oct 30 21:42:26.347453 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.347440 2579 scope.go:122] "RemoveContainer" containerID="776af6e1d06fca5704fb1ad316ab71a6b907f18fa099353dd14c5f266a71e085" Oct 30 21:42:26.347660 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.347639 2579 scope.go:122] "RemoveContainer" containerID="a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4" Oct 30 21:42:26.347861 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:26.347838 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q69rp_openshift-console-operator(d0ee0cf5-2de4-493c-85fe-f97c527d5635)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" podUID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" Oct 30 21:42:26.350228 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.350209 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" event={"ID":"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4","Type":"ContainerStarted","Data":"4084af60217f3f36b5f437d529628dfe27f705697bc50d5021fd4a77d7412890"} Oct 30 21:42:26.352629 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.352142 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" event={"ID":"f9ea4697-1a09-47e3-b6bc-ed44dae88ded","Type":"ContainerStarted","Data":"a384ea4a56d8fed954a87c76b0c5a0846b99e65a5dcf8f77c0019d8c78227fbd"} Oct 30 21:42:26.454524 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.454062 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gmcqj" podStartSLOduration=1.953230051 podStartE2EDuration="7.454048885s" totalImagesPullingTime="5.500818834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:26.453756777 +0000 UTC m=+37.889072528" watchObservedRunningTime="2026-10-30 21:42:26.454048885 +0000 UTC m=+37.889364607" Oct 30 21:42:26.685258 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.685153 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" podStartSLOduration=325.432972349 podStartE2EDuration="5m30.685134999s" totalImagesPullingTime="5.25216265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:26.568975562 +0000 UTC m=+38.004291291" watchObservedRunningTime="2026-10-30 21:42:26.685134999 +0000 UTC m=+38.120450728" Oct 30 21:42:26.799209 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.799141 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" podStartSLOduration=324.647029029 podStartE2EDuration="5m29.799123213s" totalImagesPullingTime="5.152094184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:26.798066202 +0000 UTC m=+38.233381929" watchObservedRunningTime="2026-10-30 21:42:26.799123213 +0000 UTC m=+38.234438943" Oct 30 21:42:26.923931 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:26.923881 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zxfcs" podStartSLOduration=329.836521759 podStartE2EDuration="5m34.923854913s" totalImagesPullingTime="5.087333154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:26.923026229 +0000 UTC m=+38.358341959" watchObservedRunningTime="2026-10-30 21:42:26.923854913 +0000 UTC m=+38.359170642" Oct 30 21:42:27.021815 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.020534 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m2qts"] Oct 30 21:42:27.051793 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.051223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.101325 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.101296 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m2qts"] Oct 30 21:42:27.109696 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.109657 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:42:27.109977 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.109700 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:42:27.109977 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.109744 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbz5v\"" Oct 30 21:42:27.196087 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.196057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.196237 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.196177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e1c3208-87b8-4a27-8538-6d0ddde14be7-crio-socket\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.196295 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.196260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.196295 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.196286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7z5\" (UniqueName: \"kubernetes.io/projected/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-api-access-8j7z5\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.196450 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.196432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e1c3208-87b8-4a27-8538-6d0ddde14be7-data-volume\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.249074 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.249042 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn"] Oct 30 21:42:27.277862 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.277839 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" Oct 30 21:42:27.297900 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.297874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e1c3208-87b8-4a27-8538-6d0ddde14be7-data-volume\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.298005 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.297945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.298050 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.298007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:27.298086 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.298049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e1c3208-87b8-4a27-8538-6d0ddde14be7-crio-socket\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.298123 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.298112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:27.298161 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.298150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.298199 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.298177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7z5\" (UniqueName: \"kubernetes.io/projected/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-api-access-8j7z5\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.298245 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.298204 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e1c3208-87b8-4a27-8538-6d0ddde14be7-data-volume\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.298318 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.298302 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:27.298371 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.298361 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.798343193 +0000 UTC m=+39.233658899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:27.298523 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.298481 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:27.298576 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.298553 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.298539658 +0000 UTC m=+46.733855367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:27.298633 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.298589 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:27.298633 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.298614 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.298602999 +0000 UTC m=+46.733918718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:27.298633 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.298632 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:27.798623634 +0000 UTC m=+39.233939340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : configmap "kube-rbac-proxy" not found Oct 30 21:42:27.298747 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.298730 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e1c3208-87b8-4a27-8538-6d0ddde14be7-crio-socket\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.304841 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.304820 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn"] Oct 30 21:42:27.361193 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.361116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7z5\" (UniqueName: \"kubernetes.io/projected/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-api-access-8j7z5\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.362360 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.362310 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" event={"ID":"c9a317df-0311-4ba6-bbb1-7556663b7bb5","Type":"ContainerStarted","Data":"eb9edc9047d31b3d082aac1e62e252e1f75af9a57291c5920c073e62504e9b02"} Oct 30 21:42:27.364339 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.364314 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/1.log" Oct 30 21:42:27.364954 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.364931 2579 scope.go:122] "RemoveContainer" containerID="a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4" Oct 30 21:42:27.365195 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.365171 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q69rp_openshift-console-operator(d0ee0cf5-2de4-493c-85fe-f97c527d5635)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" podUID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.398796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.398908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.398936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.398952 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.398967 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.398975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399009 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.398991708 +0000 UTC m=+46.834307422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.399030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.399069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9v74\" (UniqueName: \"kubernetes.io/projected/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96-kube-api-access-v9v74\") pod \"migrator-66df5d496b-5b5xn\" (UID: \"fe67f1be-5081-45d9-b7a5-6fbfac3c5a96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399074 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399086 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399122 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399093 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399126 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.399159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399179 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.399165131 +0000 UTC m=+46.834480856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399196 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:27.399260 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399203 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.39919361 +0000 UTC m=+46.834509349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:27.400191 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399224 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.399216532 +0000 UTC m=+46.834532241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:27.400191 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399250 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.39923441 +0000 UTC m=+46.834550123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:27.400191 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.399271 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:35.399260429 +0000 UTC m=+46.834576135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:27.410718 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.410691 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:42:27.411021 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.410970 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5cjr4\"" Oct 30 21:42:27.411021 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.411004 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:27.500813 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.500768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9v74\" (UniqueName: \"kubernetes.io/projected/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96-kube-api-access-v9v74\") pod \"migrator-66df5d496b-5b5xn\" (UID: \"fe67f1be-5081-45d9-b7a5-6fbfac3c5a96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" Oct 30 21:42:27.561106 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.561022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9v74\" (UniqueName: \"kubernetes.io/projected/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96-kube-api-access-v9v74\") pod \"migrator-66df5d496b-5b5xn\" (UID: \"fe67f1be-5081-45d9-b7a5-6fbfac3c5a96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" Oct 30 21:42:27.611836 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.611794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" Oct 30 21:42:27.629032 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.628985 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" podStartSLOduration=326.76838112 podStartE2EDuration="5m33.62897297s" totalImagesPullingTime="6.86059185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:27.628206151 +0000 UTC m=+39.063521879" watchObservedRunningTime="2026-10-30 21:42:27.62897297 +0000 UTC m=+39.064288699" Oct 30 21:42:27.804137 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.804104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.804306 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:27.804222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:27.804306 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.804239 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:27.804400 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.804313 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.804294416 +0000 UTC m=+40.239610138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : configmap "kube-rbac-proxy" not found Oct 30 21:42:27.804400 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.804378 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:27.804518 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:27.804428 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:28.804411923 +0000 UTC m=+40.239727648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:28.006285 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:28.006211 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn"] Oct 30 21:42:28.013819 ip-10-0-131-250 kubenswrapper[2579]: W1030 21:42:28.013795 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe67f1be_5081_45d9_b7a5_6fbfac3c5a96.slice/crio-3443cbb75a212e476cd1668df81a313fcb114efa545596ac80cdcdbb4802bad1 WatchSource:0}: Error finding container 3443cbb75a212e476cd1668df81a313fcb114efa545596ac80cdcdbb4802bad1: Status 404 returned error can't find the container with id 3443cbb75a212e476cd1668df81a313fcb114efa545596ac80cdcdbb4802bad1 Oct 30 21:42:28.370022 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:28.369972 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" event={"ID":"fe67f1be-5081-45d9-b7a5-6fbfac3c5a96","Type":"ContainerStarted","Data":"3443cbb75a212e476cd1668df81a313fcb114efa545596ac80cdcdbb4802bad1"} Oct 30 21:42:28.372158 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:28.371919 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sz4m5" event={"ID":"55e5bc9a-1f53-4a57-88a5-446df5df92f0","Type":"ContainerStarted","Data":"835d08cdcbe334be996fe349f72eeb78dbba408796dc6467f85e215729a9e310"} Oct 30 21:42:28.495561 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:28.495485 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sz4m5" podStartSLOduration=35.613440289 podStartE2EDuration="39.495471791s" totalImagesPullingTime="3.882031502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:28.49433833 +0000 UTC m=+39.929654057" watchObservedRunningTime="2026-10-30 21:42:28.495471791 +0000 UTC m=+39.930787520" Oct 30 21:42:28.815926 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:28.815888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:28.816099 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:28.816003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:28.816099 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:28.816023 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:28.816099 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:28.816098 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.816078053 +0000 UTC m=+42.251393772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : configmap "kube-rbac-proxy" not found Oct 30 21:42:28.816283 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:28.816168 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:28.816283 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:28.816228 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:30.816212075 +0000 UTC m=+42.251527798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:29.375888 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:29.375858 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:29.898389 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:29.898358 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:29.898587 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:29.898426 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:29.898864 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:29.898848 2579 scope.go:122] "RemoveContainer" containerID="a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4" Oct 30 21:42:29.899096 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:29.899068 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q69rp_openshift-console-operator(d0ee0cf5-2de4-493c-85fe-f97c527d5635)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" podUID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" Oct 30 21:42:30.381948 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:30.381914 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" event={"ID":"fe67f1be-5081-45d9-b7a5-6fbfac3c5a96","Type":"ContainerStarted","Data":"186bf8ca7b091f00561d5ce83836f04d02d7d10c7fd78e6f6f0224b41deefdd9"} Oct 30 21:42:30.496444 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:30.496355 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" podStartSLOduration=1.438510136 podStartE2EDuration="3.496340804s" totalImagesPullingTime="2.057830668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:30.495473082 +0000 UTC m=+41.930788812" watchObservedRunningTime="2026-10-30 21:42:30.496340804 +0000 UTC m=+41.931656532" Oct 30 21:42:30.835963 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:30.835932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:30.836122 ip-10-0-131-250 kubenswrapper[2579]: I1030 21:42:30.835998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:30.836122 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:30.836042 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:42:30.836122 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:30.836105 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.836089044 +0000 UTC m=+46.271404769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : configmap "kube-rbac-proxy" not found Oct 30 21:42:30.836122 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:30.836108 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:30.836281 ip-10-0-131-250 kubenswrapper[2579]: E1030 21:42:30.836149 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.836138649 +0000 UTC m=+46.271454354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:31.245015 ip-10-0-131-250 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:42:31.258860 ip-10-0-131-250 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:42:31.259035 ip-10-0-131-250 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:42:31.259081 ip-10-0-131-250 systemd[1]: kubelet.service: Consumed 2.941s CPU time. Oct 30 21:42:31.273943 ip-10-0-131-250 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:42:31.380831 ip-10-0-131-250 kubenswrapper[4933]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:42:31.380831 ip-10-0-131-250 kubenswrapper[4933]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:42:31.380831 ip-10-0-131-250 kubenswrapper[4933]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:42:31.380831 ip-10-0-131-250 kubenswrapper[4933]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:42:31.382810 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382741 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.382810 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382810 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382815 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382818 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382822 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382825 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382828 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382832 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382835 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382837 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382840 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382843 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382845 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382848 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382850 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382853 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382856 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382858 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382861 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382864 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382866 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.382878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382869 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382872 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382874 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382877 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382880 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382883 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382891 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382894 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382897 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382899 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382902 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382904 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382907 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382911 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382915 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382919 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382923 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382926 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382929 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.383383 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382932 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382934 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382937 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382940 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382942 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382945 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382947 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382949 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382952 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382954 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382957 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382959 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382962 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382964 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382967 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382970 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382973 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382976 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382979 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.383878 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382981 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382984 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382986 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382989 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382991 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382994 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.382997 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383000 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383002 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383005 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383007 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383010 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383012 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383015 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383017 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383020 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383022 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383024 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383027 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.384386 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383029 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383032 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383034 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383037 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383039 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383042 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383045 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383047 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383050 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383053 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383056 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383058 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383061 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383063 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383066 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383068 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383071 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383073 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383076 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383079 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.384870 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383081 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383084 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383087 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383089 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383092 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383094 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383097 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383101 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383875 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383881 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383885 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383888 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383891 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383893 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383897 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383900 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383902 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383905 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383907 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.385363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383910 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383913 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383916 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383919 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383921 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383924 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383927 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383929 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383932 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383935 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383937 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383940 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383942 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383945 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383948 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383952 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383955 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383959 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383961 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.385846 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383964 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383966 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383969 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383971 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383974 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383976 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383979 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383981 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383983 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383986 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383988 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383991 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383993 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383996 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.383998 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384001 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384003 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384006 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384008 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384011 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.386324 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384014 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384016 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384019 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384021 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384024 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384026 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384029 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384031 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384034 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384037 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384039 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384043 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384047 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384050 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384052 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384055 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384057 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384060 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384062 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384065 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.386826 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384067 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384069 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384072 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384075 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384078 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384080 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384082 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384085 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384088 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384090 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384093 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384095 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384098 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384101 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384103 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384106 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384108 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384111 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384113 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384116 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.387303 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384118 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384121 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384123 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384128 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384130 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384133 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384136 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384138 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384141 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384143 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384146 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384148 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384151 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384153 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384156 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384158 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384683 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384689 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384694 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.387802 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384697 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384701 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384704 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384706 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384709 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384712 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384715 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384718 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384720 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384723 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384726 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384728 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384731 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384734 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384737 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384742 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384744 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384747 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384749 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.388260 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384752 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384754 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384757 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384760 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384762 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384765 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384767 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384770 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384772 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384775 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384777 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384780 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384782 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384785 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384787 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384790 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384792 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384795 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384797 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384800 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.388955 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384803 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384805 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384808 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384810 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384813 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384815 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384818 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384820 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384824 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384828 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384831 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384834 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384838 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384840 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384843 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384845 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384848 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384850 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384852 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.389833 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384855 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384858 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384860 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384863 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384865 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384868 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384870 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384873 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384876 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384878 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384881 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384884 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384886 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384889 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384892 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384895 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384898 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384900 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384903 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384905 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.390681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384908 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384912 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384916 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384920 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384923 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384927 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384932 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384936 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384939 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384941 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384944 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384948 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384952 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384955 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384958 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384961 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384963 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384966 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384968 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.391626 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384971 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384973 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384976 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384978 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384981 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.384983 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385073 4933 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385081 4933 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385088 4933 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385095 4933 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385101 4933 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385105 4933 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385111 4933 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385115 4933 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385118 4933 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385121 4933 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385125 4933 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385128 4933 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385132 4933 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385135 4933 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385138 4933 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385141 4933 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385144 4933 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:42:31.392466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385147 4933 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385151 4933 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385154 4933 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385157 4933 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385160 4933 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385163 4933 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385167 4933 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385170 4933 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385173 4933 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385177 4933 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385180 4933 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385183 4933 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385185 4933 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385189 4933 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385191 4933 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385196 4933 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385198 4933 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385202 4933 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385205 4933 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385208 4933 flags.go:64] FLAG: --enable-server="true" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385211 4933 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385214 4933 flags.go:64] FLAG: --event-burst="100" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385217 4933 flags.go:64] FLAG: --event-qps="50" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385220 4933 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:42:31.393474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385223 4933 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385226 4933 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385230 4933 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385233 4933 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385236 4933 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385239 4933 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385241 4933 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385244 4933 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385247 4933 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385250 4933 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385253 4933 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385255 4933 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385258 4933 flags.go:64] FLAG: --feature-gates="" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385262 4933 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385265 4933 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385268 4933 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385271 4933 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385274 4933 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385277 4933 flags.go:64] FLAG: --help="false" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385279 4933 flags.go:64] FLAG: --hostname-override="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385282 4933 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385285 4933 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385288 4933 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:42:31.394461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385291 4933 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385294 4933 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385300 4933 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385306 4933 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385309 4933 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385311 4933 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385314 4933 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385317 4933 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385320 4933 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385323 4933 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385326 4933 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385329 4933 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385332 4933 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385334 4933 flags.go:64] FLAG: --lock-file="" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385337 4933 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385340 4933 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385343 4933 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385348 4933 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385351 4933 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385355 4933 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385359 4933 flags.go:64] FLAG: --logging-format="text" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385362 4933 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385365 4933 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385368 4933 flags.go:64] FLAG: --manifest-url="" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385371 4933 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:42:31.395407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385375 4933 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385378 4933 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385382 4933 flags.go:64] FLAG: --max-pods="110" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385385 4933 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385388 4933 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385391 4933 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385394 4933 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385396 4933 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385399 4933 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385407 4933 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385413 4933 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385417 4933 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385420 4933 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385423 4933 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385426 4933 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385429 4933 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385432 4933 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385434 4933 flags.go:64] FLAG: --port="10250" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385437 4933 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385440 4933 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-001d43e3baee20010" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385443 4933 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385446 4933 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385449 4933 flags.go:64] FLAG: --register-node="true" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385452 4933 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385456 4933 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385458 4933 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:42:31.396611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385463 4933 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385466 4933 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385470 4933 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385473 4933 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385475 4933 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385478 4933 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385481 4933 flags.go:64] FLAG: --runonce="false" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385484 4933 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385500 4933 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385503 4933 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385506 4933 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385509 4933 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385512 4933 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385515 4933 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385518 4933 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385522 4933 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385525 4933 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385527 4933 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385531 4933 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385534 4933 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385537 4933 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385539 4933 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385545 4933 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385547 4933 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385550 4933 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:42:31.397721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385554 4933 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385557 4933 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385560 4933 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385563 4933 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385566 4933 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385568 4933 flags.go:64] FLAG: --v="2" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385572 4933 flags.go:64] FLAG: --version="false" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385578 4933 flags.go:64] FLAG: --vmodule="" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385582 4933 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.385585 4933 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385711 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385716 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385720 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385723 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385726 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385729 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385731 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385734 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385736 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385739 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385741 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.398799 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385744 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385748 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385750 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385753 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385756 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385758 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385761 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385764 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385766 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385769 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385771 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385774 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385777 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385779 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385782 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385784 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385787 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385789 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385793 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385796 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.399376 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385798 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385801 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385803 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385806 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385808 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385811 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385813 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385816 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385818 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385821 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385823 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385826 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385828 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385832 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385835 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385837 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385840 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385842 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385845 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.400103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385848 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385851 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385854 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385856 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385859 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385862 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385864 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385866 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385869 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385871 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385874 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385878 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385881 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385883 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385885 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385888 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385891 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385893 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385896 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.400714 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385898 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385901 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385903 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385907 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385910 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385914 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385916 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385921 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385923 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385926 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385928 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385931 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385933 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385936 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385939 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385941 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385944 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385946 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385949 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.401210 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385951 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385954 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385956 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385959 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385961 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385965 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385968 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385970 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385973 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385975 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385978 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385980 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385983 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385985 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385988 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385991 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385993 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.401730 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.385995 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.386022 4933 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386055 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386061 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386064 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386066 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386069 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386073 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386076 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386079 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386082 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386084 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386087 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386089 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386092 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.402290 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386095 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386098 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386100 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386103 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386105 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386108 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386111 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386114 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386116 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386119 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386121 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386124 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386127 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386129 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386132 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386135 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386137 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386140 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386142 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386145 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.402753 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386148 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386151 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386153 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386156 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386158 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386161 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386163 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386166 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386169 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386171 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386174 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386176 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386179 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386181 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386184 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386186 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386189 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386191 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386197 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.403244 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386199 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386202 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386204 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386206 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386209 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386212 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386215 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386217 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386220 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386222 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386225 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386227 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386230 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386234 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386236 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386239 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386241 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386244 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386246 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386249 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.403727 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386251 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386254 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386257 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386259 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386263 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386266 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386269 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386271 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386274 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386276 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386279 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386282 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386285 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386287 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386290 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386293 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386295 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386298 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386300 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.404211 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386303 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386305 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386308 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386310 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386313 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386315 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386319 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386321 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386324 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386326 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386329 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386332 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386335 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386338 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.386341 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.390908 4933 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.390927 4933 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.390952 4933 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.390960 4933 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391045 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.404701 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391052 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391058 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391062 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391067 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391071 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391076 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391080 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391085 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391089 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391093 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391097 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391103 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391107 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391111 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391115 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391120 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391124 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391129 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391133 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.405190 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391137 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391142 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391146 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391150 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391155 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391161 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391170 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391176 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391181 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391185 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391191 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391198 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391202 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391207 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391211 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391216 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391220 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391224 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391229 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.405707 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391233 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391238 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391242 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391246 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391250 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391254 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391260 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391265 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391269 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391273 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391278 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391282 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391286 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391290 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391295 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391299 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391304 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391308 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391312 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391317 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.406160 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391321 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391325 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391329 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391333 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391338 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391342 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391347 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391351 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391355 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391359 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391363 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391367 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391371 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391375 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391379 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391383 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391388 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391392 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391397 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391402 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.406664 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391406 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391410 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391414 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391418 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391422 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391427 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391431 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391435 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391440 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391444 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391448 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391452 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391456 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391460 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391465 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391468 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391472 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391477 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391481 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391512 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.407175 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391518 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391522 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391526 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391530 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391534 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391538 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391542 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.391586 4933 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391811 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391820 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391824 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391829 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391835 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391839 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391844 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391848 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.407689 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391852 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391856 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391860 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391865 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391869 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391873 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391878 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391885 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391892 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391897 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391901 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391906 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391910 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391915 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391919 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391923 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391927 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391932 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391936 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.408216 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391940 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391944 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391948 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391952 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391956 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391961 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391965 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391969 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391973 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391977 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391982 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391986 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391991 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391994 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.391999 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392003 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392007 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392011 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392016 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392020 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.408847 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392024 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392028 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392032 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392036 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392040 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392045 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392049 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392053 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392057 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392061 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392065 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392069 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392074 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392078 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392083 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392087 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392091 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392095 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392099 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392103 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.409323 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392107 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392112 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392116 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392120 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392125 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392129 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392133 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392138 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392142 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392146 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392150 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392154 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392158 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392162 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392166 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392170 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392175 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392179 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392187 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392192 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.409860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392196 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392200 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392204 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392208 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392212 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392217 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392221 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392225 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392229 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392234 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392238 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392244 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392252 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392258 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392262 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392276 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392282 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392287 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.410402 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392291 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.392330 4933 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392381 4933 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392390 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392395 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392400 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392405 4933 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392409 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392413 4933 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392418 4933 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392422 4933 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392428 4933 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392434 4933 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392442 4933 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392447 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:42:31.410851 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392452 4933 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392456 4933 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392460 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392464 4933 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392468 4933 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392472 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392476 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392480 4933 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392484 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392507 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392512 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392516 4933 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392520 4933 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392524 4933 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392528 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392532 4933 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392537 4933 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392542 4933 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392546 4933 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392552 4933 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:42:31.411228 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392557 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392561 4933 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392565 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392570 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392574 4933 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392579 4933 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392583 4933 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392587 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392592 4933 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392596 4933 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392601 4933 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392605 4933 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392609 4933 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392613 4933 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392617 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392624 4933 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392628 4933 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392633 4933 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392637 4933 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:42:31.411761 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392644 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392648 4933 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392652 4933 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392656 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392660 4933 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392664 4933 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392668 4933 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392673 4933 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392677 4933 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392681 4933 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392687 4933 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392691 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392695 4933 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392699 4933 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392703 4933 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392707 4933 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392712 4933 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392716 4933 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392720 4933 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392725 4933 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:42:31.412219 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392729 4933 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392733 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392737 4933 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392742 4933 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392746 4933 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392750 4933 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392755 4933 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392759 4933 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392766 4933 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392770 4933 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392774 4933 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392778 4933 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392782 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392786 4933 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392790 4933 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392794 4933 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392798 4933 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392804 4933 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392809 4933 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392813 4933 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:42:31.412986 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392817 4933 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392821 4933 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392825 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392830 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392834 4933 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392838 4933 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392842 4933 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392846 4933 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392850 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392854 4933 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392858 4933 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392862 4933 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392866 4933 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:31.392870 4933 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.392948 4933 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.395698 4933 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.395770 4933 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.396051 4933 server.go:1014] "Starting client certificate rotation" Oct 30 21:42:31.413619 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.396184 4933 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.396216 4933 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:49 +0000 UTC" deadline="2028-07-27 08:00:35.06067563 +0000 UTC" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.396249 4933 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15250h18m3.664428173s" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.396779 4933 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.398516 4933 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.399973 4933 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.401607 4933 log.go:25] "Validated CRI v1 image API" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.402355 4933 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:42:31.414325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.404091 4933 fs.go:135] Filesystem UUIDs: map[0a34cad0-2997-4f5b-b98e-4101ebf0099a:/dev/nvme0n1p3 2f31de20-e1a4-47c3-932a-be00f043c906:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 30 21:42:31.414724 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.404109 4933 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d5571934de525b4caa88fda71a31989531babf03a13f2c59f86850e95ca0e03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d5571934de525b4caa88fda71a31989531babf03a13f2c59f86850e95ca0e03/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e9b2fdaabbdc1d28a20c477974601a9d6327be98dba7d1dc58fbe8f334834a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e9b2fdaabbdc1d28a20c477974601a9d6327be98dba7d1dc58fbe8f334834a1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/231a6f2bbe3c68df30cba0c1fd683ae23b63efe5abc26da7c72024fd81c236b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/231a6f2bbe3c68df30cba0c1fd683ae23b63efe5abc26da7c72024fd81c236b2/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/258afb86fd7380338c27257dffdb7e9382ebe0a158a2371be326e3021a1f057d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/258afb86fd7380338c27257dffdb7e9382ebe0a158a2371be326e3021a1f057d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3443cbb75a212e476cd1668df81a313fcb114efa545596ac80cdcdbb4802bad1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3443cbb75a212e476cd1668df81a313fcb114efa545596ac80cdcdbb4802bad1/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35a10c0396b671cee440376eb603bb60139357665a2765130d33402f25d3d2a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35a10c0396b671cee440376eb603bb60139357665a2765130d33402f25d3d2a0/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/395f2ead7c0dda3ecb0cbbcd857bfad1dd0aff5748136c3d1d4b9e5fba58cd48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/395f2ead7c0dda3ecb0cbbcd857bfad1dd0aff5748136c3d1d4b9e5fba58cd48/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4aeeaa3fa7a1ea99ff08721f2b363e8c981ac7d066eb0e8f4f0b8f2231be9a86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4aeeaa3fa7a1ea99ff08721f2b363e8c981ac7d066eb0e8f4f0b8f2231be9a86/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c84179f08afd6dea6fa37fbfe373e467b4d299a6efb1b4cc33c719688f76c86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c84179f08afd6dea6fa37fbfe373e467b4d299a6efb1b4cc33c719688f76c86/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73f6307ec4bbb6b25369f79d30de80a63f9ac59d5da2dfc8638800adc1b005c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73f6307ec4bbb6b25369f79d30de80a63f9ac59d5da2dfc8638800adc1b005c0/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82e57fc8e5eec381f62314957def4f0e1b865b7151c898fb9527dfb656494734/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82e57fc8e5eec381f62314957def4f0e1b865b7151c898fb9527dfb656494734/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8abe5c46b8139fd38061759341005be3acf9fac5347f1d30e7caf27a4580fb42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8abe5c46b8139fd38061759341005be3acf9fac5347f1d30e7caf27a4580fb42/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b29de7f55f2b2bc89c23f0d89097a922ff5e7fce245d28e3ecd1be3e58bf3cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b29de7f55f2b2bc89c23f0d89097a922ff5e7fce245d28e3ecd1be3e58bf3cc/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90e6dae0754dbefceaef28d17c2308a64bf4f064a4636cecce6cfc764c7abf91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90e6dae0754dbefceaef28d17c2308a64bf4f064a4636cecce6cfc764c7abf91/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adcbbb90aec8bc038ff107c3fcf2c08ad57cc6b7b1e6f03b7af381b831499d9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adcbbb90aec8bc038ff107c3fcf2c08ad57cc6b7b1e6f03b7af381b831499d9d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6cc4b1ddd7a49d3c43e4ab52b41de21c708e509c7d62e95af84a80ceaed918b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6cc4b1ddd7a49d3c43e4ab52b41de21c708e509c7d62e95af84a80ceaed918b/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5b9473052eeb2b8a8b3ae42fb92d746b4c4deedaa0d67b26f2362130f42ecfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5b9473052eeb2b8a8b3ae42fb92d746b4c4deedaa0d67b26f2362130f42ecfc/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f14f955699ec5b9843748aa7672231f820097cae6cdc98b75ace3e4b09bc6a23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f14f955699ec5b9843748aa7672231f820097cae6cdc98b75ace3e4b09bc6a23/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3b912f34e393b6b3180d7ef5ffbc7d1310437a7a431f9f1cf15d661f3636f5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3b912f34e393b6b3180d7ef5ffbc7d1310437a7a431f9f1cf15d661f3636f5b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8dc369c7cb6c394f7c43a41b51a09160b673b8530af9fac486e234c92a09821/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8dc369c7cb6c394f7c43a41b51a09160b673b8530af9fac486e234c92a09821/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd8c3848a20067531f25b3224285e016af80ec596ce8e37162b777b4999bf0ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd8c3848a20067531f25b3224285e016af80ec596ce8e37162b777b4999bf0ef/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~projected/kube-api-access-dvl8j:{mountpoint:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~projected/kube-api-access-dvl8j major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/061b0a6d-64e8-4a45-8ada-59d74f6023a2/volumes/kubernetes.io~projected/kube-api-access-8kkzt:{mountpoint:/var/lib/kubelet/pods/061b0a6d-64e8-4a45-8ada-59d74f6023a2/volumes/kubernetes.io~projected/kube-api-access-8kkzt major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/061b0a6d-64e8-4a45-8ada-59d74f6023a2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/061b0a6d-64e8-4a45-8ada-59d74f6023a2/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4/volumes/kubernetes.io~projected/kube-api-access-4r5qk:{mountpoint:/var/lib/kubelet/pods/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4/volumes/kubernetes.io~projected/kube-api-access-4r5qk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f607097-ece8-4833-8f34-361ac77ca2ee/volumes/kubernetes.io~projected/kube-api-access-vcs4q:{mountpoint:/var/lib/kubelet/pods/1f607097-ece8-4833-8f34-361ac77ca2ee/volumes/kubernetes.io~projected/kube-api-access-vcs4q major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f607097-ece8-4833-8f34-361ac77ca2ee/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1f607097-ece8-4833-8f34-361ac77ca2ee/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b72356e-8b7b-4392-859e-5ac980f759ef/volumes/kubernetes.io~projected/kube-api-access-xhq24:{mountpoint:/var/lib/kubelet/pods/3b72356e-8b7b-4392-859e-5ac980f759ef/volumes/kubernetes.io~projected/kube-api-access-xhq24 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~projected/kube-api-access-7p6rh:{mountpoint:/var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~projected/kube-api-access-7p6rh major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~secret/default-certificate major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5214b527-3150-40e4-9162-f712d602c83a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5214b527-3150-40e4-9162-f712d602c83a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55e5bc9a-1f53-4a57-88a5-446df5df92f0/volumes/kubernetes.io~projected/kube-api-access-8v7rp:{mountpoint:/var/lib/kubelet/pods/55e5bc9a-1f53-4a57-88a5-446df5df92f0/volumes/kubernetes.io~projected/kube-api-access-8v7rp major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e1c3208-87b8-4a27-8538-6d0ddde14be7/volumes/kubernetes.io~projected/kube-api-access-8j7z5:{mountpoint:/var/lib/kubelet/pods/5e1c3208-87b8-4a27-8538-6d0ddde14be7/volumes/kubernetes.io~projected/kube-api-access-8j7z5 major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~projected/kube-api-access-27dtl:{mountpoint:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~projected/kube-api-access-27dtl major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a4f783d-df3d-4c74-939e-22bf246e9738/volumes/kubernetes.io~projected/kube-api-access-w5n7q:{mountpoint:/var/lib/kubelet/pods/6a4f783d-df3d-4c74-939e-22bf246e9738/volumes/kubernetes.io~projected/kube-api-access-w5n7q major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~projected/kube-api-access-n6dfl:{mountpoint:/var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~projected/kube-api-access-n6dfl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dd70011-6085-49a8-b665-cb33157874fe/volumes/kubernetes.io~projected/kube-api-access-pf557:{mountpoint:/var/lib/kubelet/pods/7dd70011-6085-49a8-b665-cb33157874fe/volumes/kubernetes.io~projected/kube-api-access-pf557 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b8dc362-93da-466a-b4bd-7d33a25a00e4/volumes/kubernetes.io~projected/kube-api-access-xmkg7:{mountpoint:/var/lib/kubelet/pods/8b8dc362-93da-466a-b4bd-7d33a25a00e4/volumes/kubernetes.io~projected/kube-api-access-xmkg7 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95ce4f09-438e-4534-aeae-d427c599641a/volumes/kubernetes.io~projected/kube-api-access-zsvd5:{mountpoint:/var/lib/kubelet/pods/95ce4f09-438e-4534-aeae-d427c599641a/volumes/kubernetes.io~projected/kube-api-access-zsvd5 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7/volumes/kubernetes.io~projected/kube-api-access-w7m6w:{mountpoint:/var/lib/kubelet/pods/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7/volumes/kubernetes.io~projected/kube-api-access-w7m6w major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b10b2b7a-ecaf-434a-8248-0bdfd7df1667/volumes/kubernetes.io~projected/kube-api-access-zrjnd:{mountpoint:/var/lib/kubelet/pods/b10b2b7a-ecaf-434a-8248-0bdfd7df1667/volumes/kubernetes.io~projected/kube-api-access-zrjnd major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6780389-4c8f-449a-a007-8682974061cb/volumes/kubernetes.io~projected/kube-api-access-spc6h:{mountpoint:/var/lib/kubelet/pods/b6780389-4c8f-449a-a007-8682974061cb/volumes/kubernetes.io~projected/kube-api-access-spc6h major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a1d58f-d414-42c1-8505-79acb178f8cf/volumes/kubernetes.io~projected/kube-api-access-xc7bz:{mountpoint:/var/lib/kubelet/pods/c0a1d58f-d414-42c1-8505-79acb178f8cf/volumes/kubernetes.io~projected/kube-api-access-xc7bz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9a317df-0311-4ba6-bbb1-7556663b7bb5/volumes/kubernetes.io~projected/kube-api-access-dvrzf:{mountpoint:/var/lib/kubelet/pods/c9a317df-0311-4ba6-bbb1-7556663b7bb5/volumes/kubernetes.io~projected/kube-api-access-dvrzf major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cde9eb3a-08fa-49c1-bd82-38b414b0235f/volumes/kubernetes.io~projected/kube-api-access-5bhd6:{mountpoint:/var/lib/kubelet/pods/cde9eb3a-08fa-49c1-bd82-38b414b0235f/volumes/kubernetes.io~projected/kube-api-access-5bhd6 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ee0cf5-2de4-493c-85fe-f97c527d5635/volumes/kubernetes.io~projected/kube-api-access-54gjp:{mountpoint:/var/lib/kubelet/pods/d0ee0cf5-2de4-493c-85fe-f97c527d5635/volumes/kubernetes.io~projected/kube-api-access-54gjp major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ee0cf5-2de4-493c-85fe-f97c527d5635/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d0ee0cf5-2de4-493c-85fe-f97c527d5635/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df4f0be9-a14d-4120-a10f-c815830aef30/volumes/kubernetes.io~projected/kube-api-access-kjt2t:{mountpoint:/var/lib/kubelet/pods/df4f0be9-a14d-4120-a10f-c815830aef30/volumes/kubernetes.io~projected/kube-api-access-kjt2t major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef109382-58f3-45a9-bbd6-ef811dafc3d1/volumes/kubernetes.io~projected/kube-api-access-5g22b:{mountpoint:/var/lib/kubelet/pods/ef109382-58f3-45a9-bbd6-ef811dafc3d1/volumes/kubernetes.io~projected/kube-api-access-5g22b major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f22346-7f81-4d27-92b3-fde049d58d93/volumes/kubernetes.io~projected/kube-api-access-m75ql:{mountpoint:/var/lib/kubelet/pods/f0f22346-7f81-4d27-92b3-fde049d58d93/volumes/kubernetes.io~projected/kube-api-access-m75ql major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9/volumes/kubernetes.io~projected/kube-api-access-lhmd7:{mountpoint:/var/lib/kubelet/pods/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9/volumes/kubernetes.io~projected/kube-api-access-lhmd7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9ea4697-1a09-47e3-b6bc-ed44dae88ded/volumes/kubernetes.io~projected/kube-api-access-z7mng:{mountpoint:/var/lib/kubelet/pods/f9ea4697-1a09-47e3-b6bc-ed44dae88ded/volumes/kubernetes.io~projected/kube-api-access-z7mng major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9ea4697-1a09-47e3-b6bc-ed44dae88ded/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f9ea4697-1a09-47e3-b6bc-ed44dae88ded/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volumes/kubernetes.io~projected/kube-api-access-br58m:{mountpoint:/var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volumes/kubernetes.io~projected/kube-api-access-br58m major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96/volumes/kubernetes.io~projected/kube-api-access-v9v74:{mountpoint:/var/lib/kubelet/pods/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96/volumes/kubernetes.io~projected/kube-api-access-v9v74 major:0 minor:335 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2de880ed5f8533464ceea7953f124aff1753d552d35962f1ec5aa1e8ee7f433e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/71b356210e19c5a0a0ecea02d0e1cfc30291214bfcad8fe49047aa24a5cf950c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/584403cf0d130bc5faad2af0446eee5047a51154e62c0a3958ff441d9132a49e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7fb5ef11a886713edd983c7c381913c7dbf0e7b7515ae2e36f35554b1c3eb9d5/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b35377f132b219290f0a27d41117b122ab92b9fb81767bf041a81c05bf20de45/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5664b3d558304ab0b043d543c30b1d6a83890142693dc5e31756dd7187bd5b37/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0d0d251d3e8a48477089b14d98deb0c71a741d6173095a98d57fc8940ed820e8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3e3141ae3b9702d96ea5ee62d53fe5aec92b966220a400d6bf427cdac5001ed4/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/455242ad12318dc44e10d041aa96710bbda7b9213622c3b593624dafde1b6447/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/7e23303f43c18d1717cbbd45091126f7369031eeec7bb95c9d42f67b663d1d3c/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/4b43df9a65f228af755ad599d76098373819cfc6d74df81f15af4199c15fb1dd/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/5d741d10762381260cdab143478484f8b0d758623695f61915b0cedd5c3be90a/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1a3ed32da533f5d69e3241f5c00142969f6a2796316d60d61c945a49bd2e801c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b96366e8fb39e687991f769d1ad2b5f749bfbeb2a1761febc7fa0710cfff460a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ad4c85b7ebf88c091677da71ff84c7849df1da7bb93022203b8ee71e7038e73b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2404af6ca2fc828c966619505eed9b6bbf6d1d23db91ba9fcc9195519590e24a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/73b50a4d616c7165f953093f82a29b92fa2cad99da13d96fa131c688009f3210/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d3a9e26369fc3ada34ec6a0a2606c2ee787402e9cd6bd5b8785cbfc343181672/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8029b03df78d1383775323b3b2f295b3cba7a645ef26537543767552c7759300/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d2ec6b0ea62e07f420629ef65fd6dbe74dc246bf25a6a0d185bec46b8790974e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/673d902f56a723c2e3cc94a99a5e26bea473286ad6c28d2cd79297658dd1400d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/28cd1d8ba53101bc463d28ca936b7a307aac18b431c17de7ae0abde65f682de2/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8156791b5b696e8b2a743c11b8492c11aba94481b09266d03553e9d60bcb2218/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/4a0c1e8ba786a41141df5759fcd39f0e8af2fae3816b20eddddfee3a38a89019/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f0db38f71f59f23cc8c4c90928329b936fd558a9a51f6e1944a88c064e32f5ec/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/c22ff31600ce13eb7cac2f69e4b186c409782556d36ead10f205d13efe01dc7c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/0fb4fa3dd046af312fa946c0703e7d03b206256d1c315d610017deb654d20774/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/6abccb5960d9b68c1bd6723e30c4d583e27a4a796bb3948b9014021df403818b/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/250f90863d5892bfe6caeaaf2f9457fe66e2fbb86cef01ed956475d45dcc5ddc/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/282e5531e500683cdd628ba49b2c2411d1d0300dfb516563baf9cae57b529444/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/8392afb192934dc8111e681db70b842ad946a7ee644b93da419b4b5fdce18b08/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/fd2e094f09ff07bf4aef1808480c0fd944498b264da1b6eb27e30581571f3e9b/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/6643c0e6fbe76b8ee6350bf419ca3a26f544f4327bd968d940e009d9930da776/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/099ebe78165e20382f4b2835164b0a13b2a5f76ecdb1b0f2da6482d4cee16754/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/d252799f5530006fe5a7df3cc67d3c44564adfb0f0b821cc05c769fb78edc926/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/8ed33d5d95f8e9aa1ce7705ce8135645f1234a02a8340dd8e7c6b17596e55627/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/8c5f61e6b5abfe217faf57f08f7499b0369657450b63acd5a40f4e359ade62cb/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/b6d7c6ff605b97fed4326b9c545d7e08248de84fd54617048907bc5c2a04c14d/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/de9b1d9430f7ce322c50ab76bc0294e68cf2a8c61f9b0be2b435fe6274d142d3/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/97932072016d9b9f2c5a3ec12622908a85b91cbecdd9e6603cefc1d004241bb7/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/6323628361cf0ce603d6d7c1c73d5ce0eedf8d44e49d618c7ac799633fac84f2/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/2ea0aa43ab8617283a85bf7afae5187aac7dd133ce0b6823b63d290543480879/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/9bd0080f69340ff0331a84d75c619b493917d8f224ec8fdf337bf3ccfedf4ea0/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/8e3e673c86839bdc04cacb6049a3bc8bd429858f6f71f42369ec15a3509a2d16/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/55298f0b9c2033cb9cbd325b63a6e9d726a52b970fb4ad4c0a068b4dbbf74f1e/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/3737c5b95062646955089f78150edf1c860eba3a068f260e2904f0042ab6a7f3/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8e8815986f2606e0c7f2da468d814febb652128690473547f787e45a58bdbf1f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6c3128cfb729700f5774e6e2846806ed77305f39e97b8bdcb6a5140cf2f6a0d7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b245ef4645e2fee09d33172326b0b50b2c056647f0d1aab3cc663f22f1994aa7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/61ea68acefdf16496e31bb84aa68ea81e053f109b995d571d8a446504546de43/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/00a151cfdea4491c46546071fc5f54176fc6c120c60dde822d249c0d015b856d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2447769ad00e8a165dbfae8fc6e4ce77b6fa82019f3f7d471f56228cc121c8ec/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/17821b7ccccfda0a5ce40f8be263c6051ea87b0338924af07595cebbc9823ecb/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/068136e6430073da2492738920e17b585ff4c9a9aef960f68f6bf1a43f1e241c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/35d3b75a4fbca6607f499d0272e8b4e9afc78aa672a266d8267cdbdb7aade45d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7a6a8166bebd746b720de712f50999b0b1beac4e1222d740312c14914c2154d5/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.411891 4933 manager.go:217] Machine: {Timestamp:2026-10-30 21:42:31.411410075 +0000 UTC m=+0.092559960 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099475 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27aa86fca7ff3ca2009b398f0ba360 SystemUUID:ec27aa86-fca7-ff3c-a200-9b398f0ba360 BootID:cbee68c7-ed4c-45ec-abeb-0c55525242ee Filesystems:[{Device:/var/lib/kubelet/pods/d0ee0cf5-2de4-493c-85fe-f97c527d5635/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8abe5c46b8139fd38061759341005be3acf9fac5347f1d30e7caf27a4580fb42/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0ee0cf5-2de4-493c-85fe-f97c527d5635/volumes/kubernetes.io~projected/kube-api-access-54gjp DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f22346-7f81-4d27-92b3-fde049d58d93/volumes/kubernetes.io~projected/kube-api-access-m75ql DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f14f955699ec5b9843748aa7672231f820097cae6cdc98b75ace3e4b09bc6a23/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e9b2fdaabbdc1d28a20c477974601a9d6327be98dba7d1dc58fbe8f334834a1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7/volumes/kubernetes.io~projected/kube-api-access-w7m6w DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35a10c0396b671cee440376eb603bb60139357665a2765130d33402f25d3d2a0/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/258afb86fd7380338c27257dffdb7e9382ebe0a158a2371be326e3021a1f057d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82e57fc8e5eec381f62314957def4f0e1b865b7151c898fb9527dfb656494734/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5214b527-3150-40e4-9162-f712d602c83a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4aeeaa3fa7a1ea99ff08721f2b363e8c981ac7d066eb0e8f4f0b8f2231be9a86/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b10b2b7a-ecaf-434a-8248-0bdfd7df1667/volumes/kubernetes.io~projected/kube-api-access-zrjnd DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/061b0a6d-64e8-4a45-8ada-59d74f6023a2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90e6dae0754dbefceaef28d17c2308a64bf4f064a4636cecce6cfc764c7abf91/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e1c3208-87b8-4a27-8538-6d0ddde14be7/volumes/kubernetes.io~projected/kube-api-access-8j7z5 DeviceMajor:0 DeviceMinor:316 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/395f2ead7c0dda3ecb0cbbcd857bfad1dd0aff5748136c3d1d4b9e5fba58cd48/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cde9eb3a-08fa-49c1-bd82-38b414b0235f/volumes/kubernetes.io~projected/kube-api-access-5bhd6 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4/volumes/kubernetes.io~projected/kube-api-access-4r5qk DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9ea4697-1a09-47e3-b6bc-ed44dae88ded/volumes/kubernetes.io~projected/kube-api-access-z7mng DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a4f783d-df3d-4c74-939e-22bf246e9738/volumes/kubernetes.io~projected/kube-api-access-w5n7q DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd8c3848a20067531f25b3224285e016af80ec596ce8e37162b777b4999bf0ef/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/061b0a6d-64e8-4a45-8ada-59d74f6023a2/volumes/kubernetes.io~projected/kube-api-access-8kkzt DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~projected/kube-api-access-n6dfl DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b5b5c38-04af-42ba-9845-41520e647cbe/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df4f0be9-a14d-4120-a10f-c815830aef30/volumes/kubernetes.io~projected/kube-api-access-kjt2t DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a1d58f-d414-42c1-8505-79acb178f8cf/volumes/kubernetes.io~projected/kube-api-access-xc7bz DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adcbbb90aec8bc038ff107c3fcf2c08ad57cc6b7b1e6f03b7af381b831499d9d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~projected/kube-api-access-7p6rh DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/95ce4f09-438e-4534-aeae-d427c599641a/volumes/kubernetes.io~projected/kube-api-access-zsvd5 DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6780389-4c8f-449a-a007-8682974061cb/volumes/kubernetes.io~projected/kube-api-access-spc6h DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3b912f34e393b6b3180d7ef5ffbc7d1310437a7a431f9f1cf15d661f3636f5b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3be119ed-6228-46e8-a71b-8f7fb6e812f8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d5571934de525b4caa88fda71a31989531babf03a13f2c59f86850e95ca0e03/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6cc4b1ddd7a49d3c43e4ab52b41de21c708e509c7d62e95af84a80ceaed918b/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3443cbb75a212e476cd1668df81a313fcb114efa545596ac80cdcdbb4802bad1/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8dc369c7cb6c394f7c43a41b51a09160b673b8530af9fac486e234c92a09821/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1f607097-ece8-4833-8f34-361ac77ca2ee/volumes/kubernetes.io~projected/kube-api-access-vcs4q DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c9a317df-0311-4ba6-bbb1-7556663b7bb5/volumes/kubernetes.io~projected/kube-api-access-dvrzf DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c84179f08afd6dea6fa37fbfe373e467b4d299a6efb1b4cc33c719688f76c86/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3b72356e-8b7b-4392-859e-5ac980f759ef/volumes/kubernetes.io~projected/kube-api-access-xhq24 DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b29de7f55f2b2bc89c23f0d89097a922ff5e7fce245d28e3ecd1be3e58bf3cc/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/55e5bc9a-1f53-4a57-88a5-446df5df92f0/volumes/kubernetes.io~projected/kube-api-access-8v7rp DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5b9473052eeb2b8a8b3ae42fb92d746b4c4deedaa0d67b26f2362130f42ecfc/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f607097-ece8-4833-8f34-361ac77ca2ee/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73f6307ec4bbb6b25369f79d30de80a63f9ac59d5da2dfc8638800adc1b005c0/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96/volumes/kubernetes.io~projected/kube-api-access-v9v74 DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdb00f28-52e5-48e2-a034-7193a74f8157/volumes/kubernetes.io~projected/kube-api-access-br58m DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/231a6f2bbe3c68df30cba0c1fd683ae23b63efe5abc26da7c72024fd81c236b2/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7dd70011-6085-49a8-b665-cb33157874fe/volumes/kubernetes.io~projected/kube-api-access-pf557 DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9/volumes/kubernetes.io~projected/kube-api-access-lhmd7 DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes/kubernetes.io~projected/kube-api-access-27dtl DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef109382-58f3-45a9-bbd6-ef811dafc3d1/volumes/kubernetes.io~projected/kube-api-access-5g22b DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes/kubernetes.io~projected/kube-api-access-dvl8j DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b8dc362-93da-466a-b4bd-7d33a25a00e4/volumes/kubernetes.io~projected/kube-api-access-xmkg7 DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9ea4697-1a09-47e3-b6bc-ed44dae88ded/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:231a6f2bbe3c68d MacAddress:7e:b3:9e:8c:2a:31 Speed:10000 Mtu:8901} {Name:3443cbb75a212e4 MacAddress:a2:70:46:bb:1a:96 Speed:10000 Mtu:8901} {Name:35a10c0396b671c MacAddress:62:66:4e:e5:6f:6d Speed:10000 Mtu:8901} {Name:5c84179f08afd6d MacAddress:36:74:90:b6:88:0a Speed:10000 Mtu:8901} {Name:73f6307ec4bbb6b MacAddress:26:d5:64:90:35:89 Speed:10000 Mtu:8901} {Name:82e57fc8e5eec38 MacAddress:0e:2a:3c:fb:ba:93 Speed:10000 Mtu:8901} {Name:8abe5c46b8139fd MacAddress:b2:21:df:03:0c:67 Speed:10000 Mtu:8901} {Name:8b29de7f55f2b2b MacAddress:76:e7:cd:9a:a4:e6 Speed:10000 Mtu:8901} {Name:90e6dae0754dbef MacAddress:92:0d:e2:70:40:08 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:5d:df:88:16:5d Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:b9:2d:28:6c:ab Speed:0 Mtu:8901} {Name:d6cc4b1ddd7a49d MacAddress:76:4b:44:b8:9e:20 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:5d:df:88:16:5d Speed:0 Mtu:9001} {Name:f14f955699ec5b9 MacAddress:f2:bb:d4:6c:c0:d2 Speed:10000 Mtu:8901} {Name:f8dc369c7cb6c39 MacAddress:a6:02:78:e4:f5:24 Speed:10000 Mtu:8901} {Name:fd8c3848a200675 MacAddress:52:1b:a2:9c:86:2d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2a:49:fb:b7:2e:90 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:22:b6:ae:74:2f:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412216 4933 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412253 4933 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412547 4933 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412571 4933 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-250.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412685 4933 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412693 4933 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412704 4933 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412721 4933 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412737 4933 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412890 4933 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.412996 4933 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.413033 4933 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.413272 4933 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.413388 4933 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.413408 4933 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.413421 4933 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.414650 4933 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:42:31.414972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.414676 4933 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:42:31.415805 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.415458 4933 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:42:31.417745 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.417731 4933 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:42:31.417745 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.417745 4933 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:42:31.417857 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.417848 4933 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:42:31.418020 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418010 4933 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:42:31.418050 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418040 4933 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:42:31.418134 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418125 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:42:31.418164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418139 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:42:31.418164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418145 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:42:31.418164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418151 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:42:31.418164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418157 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:42:31.418164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418162 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418180 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418186 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418192 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418197 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418209 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418219 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418242 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:42:31.418292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.418247 4933 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:42:31.421501 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.421474 4933 server.go:1263] "Started kubelet" Oct 30 21:42:31.422141 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.421638 4933 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:42:31.422141 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.421653 4933 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:42:31.422141 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.421721 4933 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:42:31.422141 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.422055 4933 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:42:31.422018 ip-10-0-131-250 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:42:31.422885 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.422865 4933 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.428076 4933 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.428102 4933 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.428156 4933 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:51 +0000 UTC" deadline="2028-07-19 13:50:53.215574456 +0000 UTC" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.428173 4933 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15064h8m21.787404376s" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.428183 4933 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.428196 4933 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.428309 4933 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:42:31.428530 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.428190 4933 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:42:31.429766 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.429746 4933 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:42:31.429865 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.429769 4933 factory.go:55] Registering systemd factory Oct 30 21:42:31.429865 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.429780 4933 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:42:31.430021 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.429989 4933 factory.go:153] Registering CRI-O factory Oct 30 21:42:31.430021 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430002 4933 factory.go:223] Registration of the crio container factory successfully Oct 30 21:42:31.430021 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430018 4933 factory.go:103] Registering Raw factory Oct 30 21:42:31.430153 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430035 4933 manager.go:1199] Started watching for new ooms in manager Oct 30 21:42:31.430397 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430374 4933 manager.go:319] Starting recovery of all containers Oct 30 21:42:31.430876 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430846 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81526e36-032f-4f22-a94e-5e48680b7bc7" volumeName="kubernetes.io/configmap/81526e36-032f-4f22-a94e-5e48680b7bc7-nginx-conf" seLinuxMountContext="" Oct 30 21:42:31.430965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430878 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f22346-7f81-4d27-92b3-fde049d58d93" volumeName="kubernetes.io/projected/f0f22346-7f81-4d27-92b3-fde049d58d93-kube-api-access-m75ql" seLinuxMountContext="" Oct 30 21:42:31.430965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430891 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ea4697-1a09-47e3-b6bc-ed44dae88ded" volumeName="kubernetes.io/secret/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-serving-cert" seLinuxMountContext="" Oct 30 21:42:31.430965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430903 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bf86a4-9e40-472e-89a5-83e099fd7953" volumeName="kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:42:31.430965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430915 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" volumeName="kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:42:31.430965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430926 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55e5bc9a-1f53-4a57-88a5-446df5df92f0" volumeName="kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp" seLinuxMountContext="" Oct 30 21:42:31.430965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430940 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf6cb3-60c7-471c-ba9e-62c236e4c7c4" volumeName="kubernetes.io/projected/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-kube-api-access-4r5qk" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430951 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" volumeName="kubernetes.io/projected/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-kube-api-access-w7m6w" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430981 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ed16df-9be5-40ee-b222-ed7c2d11455a" volumeName="kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.430992 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bf86a4-9e40-472e-89a5-83e099fd7953" volumeName="kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431005 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4f783d-df3d-4c74-939e-22bf246e9738" volumeName="kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-cni-binary-copy" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431017 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ee0cf5-2de4-493c-85fe-f97c527d5635" volumeName="kubernetes.io/projected/d0ee0cf5-2de4-493c-85fe-f97c527d5635-kube-api-access-54gjp" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431028 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ee0cf5-2de4-493c-85fe-f97c527d5635" volumeName="kubernetes.io/secret/d0ee0cf5-2de4-493c-85fe-f97c527d5635-serving-cert" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431042 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef109382-58f3-45a9-bbd6-ef811dafc3d1" volumeName="kubernetes.io/projected/ef109382-58f3-45a9-bbd6-ef811dafc3d1-kube-api-access-5g22b" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431056 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f22346-7f81-4d27-92b3-fde049d58d93" volumeName="kubernetes.io/configmap/f0f22346-7f81-4d27-92b3-fde049d58d93-serviceca" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431068 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f607097-ece8-4833-8f34-361ac77ca2ee" volumeName="kubernetes.io/projected/1f607097-ece8-4833-8f34-361ac77ca2ee-kube-api-access-vcs4q" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431080 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bf86a4-9e40-472e-89a5-83e099fd7953" volumeName="kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431092 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cde9eb3a-08fa-49c1-bd82-38b414b0235f" volumeName="kubernetes.io/configmap/cde9eb3a-08fa-49c1-bd82-38b414b0235f-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431104 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1c3208-87b8-4a27-8538-6d0ddde14be7" volumeName="kubernetes.io/empty-dir/5e1c3208-87b8-4a27-8538-6d0ddde14be7-data-volume" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431115 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4f783d-df3d-4c74-939e-22bf246e9738" volumeName="kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-daemon-config" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431126 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b8dc362-93da-466a-b4bd-7d33a25a00e4" volumeName="kubernetes.io/projected/8b8dc362-93da-466a-b4bd-7d33a25a00e4-kube-api-access-xmkg7" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431138 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3be119ed-6228-46e8-a71b-8f7fb6e812f8" volumeName="kubernetes.io/projected/3be119ed-6228-46e8-a71b-8f7fb6e812f8-kube-api-access-7p6rh" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431149 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95ce4f09-438e-4534-aeae-d427c599641a" volumeName="kubernetes.io/projected/95ce4f09-438e-4534-aeae-d427c599641a-kube-api-access-zsvd5" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431160 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a1d58f-d414-42c1-8505-79acb178f8cf" volumeName="kubernetes.io/projected/c0a1d58f-d414-42c1-8505-79acb178f8cf-kube-api-access-xc7bz" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431172 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe67f1be-5081-45d9-b7a5-6fbfac3c5a96" volumeName="kubernetes.io/projected/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96-kube-api-access-v9v74" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431194 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ed16df-9be5-40ee-b222-ed7c2d11455a" volumeName="kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431207 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bf86a4-9e40-472e-89a5-83e099fd7953" volumeName="kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431218 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdb00f28-52e5-48e2-a034-7193a74f8157" volumeName="kubernetes.io/secret/fdb00f28-52e5-48e2-a034-7193a74f8157-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431235 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf6cb3-60c7-471c-ba9e-62c236e4c7c4" volumeName="kubernetes.io/secret/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-serving-cert" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431247 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b5b5c38-04af-42ba-9845-41520e647cbe" volumeName="kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-tmp" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431258 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dd70011-6085-49a8-b665-cb33157874fe" volumeName="kubernetes.io/projected/7dd70011-6085-49a8-b665-cb33157874fe-kube-api-access-pf557" seLinuxMountContext="" Oct 30 21:42:31.431262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431269 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9a317df-0311-4ba6-bbb1-7556663b7bb5" volumeName="kubernetes.io/projected/c9a317df-0311-4ba6-bbb1-7556663b7bb5-kube-api-access-dvrzf" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431281 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ee0cf5-2de4-493c-85fe-f97c527d5635" volumeName="kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-config" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431294 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df4f0be9-a14d-4120-a10f-c815830aef30" volumeName="kubernetes.io/projected/df4f0be9-a14d-4120-a10f-c815830aef30-kube-api-access-kjt2t" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431307 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8ea0e11-f736-4db0-9ffe-d46ed7479dd9" volumeName="kubernetes.io/projected/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9-kube-api-access-lhmd7" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431318 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ea4697-1a09-47e3-b6bc-ed44dae88ded" volumeName="kubernetes.io/configmap/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-config" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431329 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ed16df-9be5-40ee-b222-ed7c2d11455a" volumeName="kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431341 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f607097-ece8-4833-8f34-361ac77ca2ee" volumeName="kubernetes.io/secret/1f607097-ece8-4833-8f34-361ac77ca2ee-serving-cert" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431352 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bf86a4-9e40-472e-89a5-83e099fd7953" volumeName="kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431363 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b8dc362-93da-466a-b4bd-7d33a25a00e4" volumeName="kubernetes.io/configmap/8b8dc362-93da-466a-b4bd-7d33a25a00e4-telemetry-config" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431375 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" volumeName="kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431388 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0adf6cb3-60c7-471c-ba9e-62c236e4c7c4" volumeName="kubernetes.io/configmap/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-config" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431399 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f607097-ece8-4833-8f34-361ac77ca2ee" volumeName="kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-snapshots" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431411 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9" volumeName="kubernetes.io/secret/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-agent-certs" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431422 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3be119ed-6228-46e8-a71b-8f7fb6e812f8" volumeName="kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-default-certificate" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431434 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bf86a4-9e40-472e-89a5-83e099fd7953" volumeName="kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431446 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b10b2b7a-ecaf-434a-8248-0bdfd7df1667" volumeName="kubernetes.io/projected/b10b2b7a-ecaf-434a-8248-0bdfd7df1667-kube-api-access-zrjnd" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431457 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ee0cf5-2de4-493c-85fe-f97c527d5635" volumeName="kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-trusted-ca" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431469 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ea4697-1a09-47e3-b6bc-ed44dae88ded" volumeName="kubernetes.io/projected/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-kube-api-access-z7mng" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431481 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ed16df-9be5-40ee-b222-ed7c2d11455a" volumeName="kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431512 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f607097-ece8-4833-8f34-361ac77ca2ee" volumeName="kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-tmp" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431524 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64bf86a4-9e40-472e-89a5-83e099fd7953" volumeName="kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431536 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b5b5c38-04af-42ba-9845-41520e647cbe" volumeName="kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-tuned" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431549 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f607097-ece8-4833-8f34-361ac77ca2ee" volumeName="kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431563 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cde9eb3a-08fa-49c1-bd82-38b414b0235f" volumeName="kubernetes.io/projected/cde9eb3a-08fa-49c1-bd82-38b414b0235f-kube-api-access-5bhd6" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431576 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdb00f28-52e5-48e2-a034-7193a74f8157" volumeName="kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-env-overrides" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431595 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3be119ed-6228-46e8-a71b-8f7fb6e812f8" volumeName="kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-stats-auth" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431607 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4f783d-df3d-4c74-939e-22bf246e9738" volumeName="kubernetes.io/projected/6a4f783d-df3d-4c74-939e-22bf246e9738-kube-api-access-w5n7q" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431620 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95ce4f09-438e-4534-aeae-d427c599641a" volumeName="kubernetes.io/configmap/95ce4f09-438e-4534-aeae-d427c599641a-config-volume" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431632 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" volumeName="kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-binary-copy" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431643 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdb00f28-52e5-48e2-a034-7193a74f8157" volumeName="kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-config" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431655 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ed16df-9be5-40ee-b222-ed7c2d11455a" volumeName="kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431666 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ed16df-9be5-40ee-b222-ed7c2d11455a" volumeName="kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431694 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b72356e-8b7b-4392-859e-5ac980f759ef" volumeName="kubernetes.io/projected/3b72356e-8b7b-4392-859e-5ac980f759ef-kube-api-access-xhq24" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431706 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95ce4f09-438e-4534-aeae-d427c599641a" volumeName="kubernetes.io/empty-dir/95ce4f09-438e-4534-aeae-d427c599641a-tmp-dir" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431720 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6780389-4c8f-449a-a007-8682974061cb" volumeName="kubernetes.io/empty-dir/b6780389-4c8f-449a-a007-8682974061cb-tmp-dir" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431733 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef109382-58f3-45a9-bbd6-ef811dafc3d1" volumeName="kubernetes.io/configmap/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-config" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431745 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5214b527-3150-40e4-9162-f712d602c83a" volumeName="kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431756 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e1c3208-87b8-4a27-8538-6d0ddde14be7" volumeName="kubernetes.io/projected/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-api-access-8j7z5" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431768 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="061b0a6d-64e8-4a45-8ada-59d74f6023a2" volumeName="kubernetes.io/projected/061b0a6d-64e8-4a45-8ada-59d74f6023a2-kube-api-access-8kkzt" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431781 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f607097-ece8-4833-8f34-361ac77ca2ee" volumeName="kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-service-ca-bundle" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431793 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9" volumeName="kubernetes.io/configmap/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-konnectivity-ca" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431804 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6780389-4c8f-449a-a007-8682974061cb" volumeName="kubernetes.io/projected/b6780389-4c8f-449a-a007-8682974061cb-kube-api-access-spc6h" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431815 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdb00f28-52e5-48e2-a034-7193a74f8157" volumeName="kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431827 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdb00f28-52e5-48e2-a034-7193a74f8157" volumeName="kubernetes.io/projected/fdb00f28-52e5-48e2-a034-7193a74f8157-kube-api-access-br58m" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431839 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04ed16df-9be5-40ee-b222-ed7c2d11455a" volumeName="kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431851 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="061b0a6d-64e8-4a45-8ada-59d74f6023a2" volumeName="kubernetes.io/secret/061b0a6d-64e8-4a45-8ada-59d74f6023a2-cert" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431862 4933 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b5b5c38-04af-42ba-9845-41520e647cbe" volumeName="kubernetes.io/projected/7b5b5c38-04af-42ba-9845-41520e647cbe-kube-api-access-n6dfl" seLinuxMountContext="" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431874 4933 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:42:31.432523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.431882 4933 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:42:31.446441 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.445417 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/1.log" Oct 30 21:42:31.449085 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.449064 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/1.log" Oct 30 21:42:31.456008 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.455978 4933 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:42:31.457851 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.457793 4933 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:42:31.457851 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.457817 4933 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:42:31.457999 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.457856 4933 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:42:31.457999 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.457900 4933 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:42:31.464074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.464049 4933 generic.go:356] "Generic (PLEG): container finished" podID="908bb8ba293781da3344be83dfda2377" containerID="4d5fb9e646c4d45e7da2f1facf040038d008dd5a78817c729a7bb4c63b260ac3" exitCode=0 Oct 30 21:42:31.473310 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.473292 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/1.log" Oct 30 21:42:31.473396 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.473318 4933 generic.go:356] "Generic (PLEG): container finished" podID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" containerID="a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4" exitCode=255 Oct 30 21:42:31.487994 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.487973 4933 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="4de00840e426ff20a510f6bcbbeed7c436283b3635fb85c7159ec2cd0e0b169c" exitCode=0 Oct 30 21:42:31.487994 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.487993 4933 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="1cc1383b9c52122ef1f7293fc98487b2ab0565491ac13ebf07354654aa0bef25" exitCode=0 Oct 30 21:42:31.488102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.487999 4933 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="11f006004e19cf2be95d68b8aeba1ec13a9b55e1b2edfa7a1c045b71290f1a75" exitCode=0 Oct 30 21:42:31.488102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.488006 4933 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="2f8be2f8f17d12c25b4ba381028e089dba66c6c2f47cdaafed7f8920c13ae242" exitCode=0 Oct 30 21:42:31.488102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.488012 4933 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="4148184c5e12347eabbc7315648e2a7ae9be59efc8f227de38951a93727e0953" exitCode=0 Oct 30 21:42:31.488102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.488017 4933 generic.go:356] "Generic (PLEG): container finished" podID="a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7" containerID="a3a6fcb9c6db2698ddd31767bf5876bb66a256d0d36aef739c1c7b771e2b67e9" exitCode=0 Oct 30 21:42:31.520908 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.520856 4933 manager.go:324] Recovery completed Oct 30 21:42:31.528386 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.528365 4933 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:42:31.539171 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.539154 4933 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:31.542474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.542461 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:31.542570 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.542506 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:31.542570 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.542521 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:31.544663 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544648 4933 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:42:31.544712 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544663 4933 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:42:31.544712 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544667 4933 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:31.544712 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544682 4933 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:42:31.544840 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544825 4933 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:42:31.544872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544842 4933 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:42:31.544872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544858 4933 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:42:31.544872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544863 4933 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:42:31.544872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544869 4933 policy_none.go:50] "Start" Oct 30 21:42:31.544968 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544878 4933 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:42:31.544968 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.544887 4933 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:42:31.545576 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.545561 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:31.545652 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.545592 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:31.545652 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.545609 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:31.553278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.553261 4933 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:42:31.553278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.553280 4933 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:42:31.553366 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.553287 4933 policy_none.go:44] "Start" Oct 30 21:42:31.557313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.557284 4933 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:42:31.557957 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.557939 4933 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:42:31.560982 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.560969 4933 manager.go:341] "Starting Device Plugin manager" Oct 30 21:42:31.561066 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.560995 4933 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:42:31.561066 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561007 4933 server.go:87] "Starting device plugin registration server" Oct 30 21:42:31.561285 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561271 4933 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:42:31.561344 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561285 4933 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:42:31.561399 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561382 4933 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:42:31.561539 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561481 4933 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:42:31.561599 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561554 4933 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:42:31.561599 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561561 4933 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:42:31.561686 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.561603 4933 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:42:31.561537182Z","UUID":"f452c3dc-768a-4d2f-8322-6cd90a79f9ab","Handler":null,"Name":"","Endpoint":""} Oct 30 21:42:31.562968 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.562943 4933 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:42:31.562968 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.562969 4933 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:42:31.563420 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.563402 4933 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:42:31.563526 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.563441 4933 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-250.us-west-1.compute.internal\" not found" Oct 30 21:42:31.574015 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.573999 4933 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:31.624060 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.624038 4933 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:42:31.624621 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.624608 4933 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:31.625218 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.625195 4933 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:31.662178 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.662157 4933 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:42:31.662949 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.662934 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:42:31.663026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.662963 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:42:31.663026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.662978 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 30 21:42:31.663026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.663024 4933 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.676006 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.675989 4933 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:42:31.758026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.757992 4933 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal"] Oct 30 21:42:31.758504 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.758461 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"908bb8ba293781da3344be83dfda2377","Type":"ContainerDied","Data":"4d5fb9e646c4d45e7da2f1facf040038d008dd5a78817c729a7bb4c63b260ac3"} Oct 30 21:42:31.758542 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.758516 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"908bb8ba293781da3344be83dfda2377","Type":"ContainerStarted","Data":"80daee1c1ce8a7a3beea614cfff0ee2e96de2326c4c0bcf62ba5bcc6a7a5d2d9"} Oct 30 21:42:31.758583 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.758575 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"801ebfba1db02fef7c7a04b1978809fc","Type":"ContainerStarted","Data":"9548578180e75180dae09454eb30122c6f8e5135becc891708f42ce6c9913517"} Oct 30 21:42:31.758623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.758592 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" event={"ID":"801ebfba1db02fef7c7a04b1978809fc","Type":"ContainerStarted","Data":"4382e77372bece20a5b878012791e1420fc975dc74a1d6c39a6d712a62a927ce"} Oct 30 21:42:31.758666 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.758629 4933 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.758718 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.758704 4933 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.773542 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.773484 4933 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.773591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.773550 4933 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.773964 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.773946 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeNotReady" Oct 30 21:42:31.774033 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.773981 4933 setters.go:546] "Node became not ready" node="ip-10-0-131-250.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:42:31Z","lastTransitionTime":"2026-10-30T21:42:31Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:42:31.791374 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.791351 4933 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-250.us-west-1.compute.internal" is owned by different node (oldNodeID="54c620b7-4678-4af9-88f9-c6bb30f13b6d", newNodeID=""), cleaning up... Oct 30 21:42:31.822536 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.822514 4933 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:31.822612 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.822556 4933 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.823316 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.823303 4933 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:31.823350 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:31.823333 4933 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.926150 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.926123 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.926242 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.926157 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:31.926242 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:31.926178 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/801ebfba1db02fef7c7a04b1978809fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"801ebfba1db02fef7c7a04b1978809fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.026655 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.026628 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.026794 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.026668 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.026794 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.026686 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/801ebfba1db02fef7c7a04b1978809fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"801ebfba1db02fef7c7a04b1978809fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.026794 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.026729 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/801ebfba1db02fef7c7a04b1978809fc-config\") pod \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"801ebfba1db02fef7c7a04b1978809fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.026794 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.026740 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.026794 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.026752 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/908bb8ba293781da3344be83dfda2377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" (UID: \"908bb8ba293781da3344be83dfda2377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.415395 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.415330 4933 apiserver.go:52] "Watching apiserver" Oct 30 21:42:32.513930 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.513909 4933 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.514067 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.514040 4933 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.571466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.571436 4933 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:42:32.572139 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.572111 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-v4lbg","openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn","openshift-insights/insights-operator-7c4986d449-zxfcs","openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f","openshift-multus/network-metrics-daemon-g7hlg","openshift-image-registry/node-ca-wqw28","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f","openshift-dns/dns-default-pb7j7","openshift-multus/multus-additional-cni-plugins-jw58x","openshift-ingress-canary/ingress-canary-gmcqj","openshift-multus/multus-vvgps","openshift-dns/node-resolver-lq586","kube-system/global-pull-secret-syncer-qdn6l","openshift-cluster-node-tuning-operator/tuned-ct72s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7","openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn","openshift-network-diagnostics/network-check-target-sz4m5","openshift-image-registry/image-registry-64975494cd-d2ggk","openshift-insights/insights-runtime-extractor-m2qts","openshift-network-console/networking-console-plugin-86c478bb88-5xpsq","kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk","kube-system/kas-connection-checker-685895c7c9-tvk4h","kube-system/kas-connection-checker-685895c7c9-dj4nw","openshift-ingress/router-default-7444c59677-nrb6q","kube-system/kas-connection-checker-685895c7c9-ptmmz","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn","openshift-network-operator/iptables-alerter-59qkx","openshift-ovn-kubernetes/ovnkube-node-vgt94","openshift-console-operator/console-operator-7cd56b8dbc-q69rp","openshift-image-registry/image-registry-675b86c7dc-q72nj"] Oct 30 21:42:32.573567 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.573549 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:32.574640 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.574621 4933 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:32.574719 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.574694 4933 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.575665 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.575632 4933 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:42:32.575762 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.575692 4933 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-250.us-west-1.compute.internal" Oct 30 21:42:32.578266 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.578238 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.579259 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.578847 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:32.579424 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.579409 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:32.579818 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.579805 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.583600 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.583583 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:32.585292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.585277 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.585761 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.585750 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.587029 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.587014 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.627199 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.627183 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.627295 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.627279 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.628395 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628366 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.628572 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628558 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:42:32.628693 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628676 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.628765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628709 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.628765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628723 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:42:32.628765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628744 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.628765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628756 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.629015 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.628997 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.629079 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629063 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.629131 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629074 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-systemd\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629131 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629107 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.629131 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629110 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-tmp\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629131 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629124 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629136 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629149 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629169 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rpk5c\"" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629195 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629218 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629214 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629244 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-conf\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629299 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-tmp\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629306 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-sys\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629373 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysconfig\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629388 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629402 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629401 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-kubernetes\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629436 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629444 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-tuned\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629460 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bz8qf\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629518 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf557\" (UniqueName: \"kubernetes.io/projected/7dd70011-6085-49a8-b665-cb33157874fe-kube-api-access-pf557\") pod \"kas-connection-checker-685895c7c9-dj4nw\" (UID: \"7dd70011-6085-49a8-b665-cb33157874fe\") " pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629536 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629559 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-tzxpl\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629611 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-serving-cert\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629623 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cmjqq\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629661 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lbgb5\"" Oct 30 21:42:32.629714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629667 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-konnectivity-ca\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629717 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-tuned\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629800 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-config\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629841 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629838 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g22b\" (UniqueName: \"kubernetes.io/projected/ef109382-58f3-45a9-bbd6-ef811dafc3d1-kube-api-access-5g22b\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.629962 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630069 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-lib-modules\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630124 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630156 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7m6w\" (UniqueName: \"kubernetes.io/projected/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-kube-api-access-w7m6w\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630206 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-system-cni-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630226 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9lfqr\"" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630239 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cnibin\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.630325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630292 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-agent-certs\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630336 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-run\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630353 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-var-lib-kubelet\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630369 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-host\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630411 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5qk\" (UniqueName: \"kubernetes.io/projected/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-kube-api-access-4r5qk\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630508 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630605 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630721 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:32.630873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.630746 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-modprobe-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.632417 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.632400 4933 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:42:32.679627 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.679581 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:42:32.680011 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.679995 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:42:32.681634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.680688 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:42:32.681634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.680735 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:42:32.681634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681033 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:42:32.681634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681205 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.681634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681268 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:42:32.681634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681469 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.681634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681537 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.681990 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681773 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:42:32.681990 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681820 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:42:32.681990 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.681980 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.682121 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.682056 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.682302 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.682286 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.682529 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.682510 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:42:32.682843 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.682822 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6xflp\"" Oct 30 21:42:32.682984 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.682969 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xcxlp\"" Oct 30 21:42:32.683057 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.682820 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:42:32.683229 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.683206 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zs9h6\"" Oct 30 21:42:32.683622 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.683603 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:42:32.686235 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.686218 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf557\" (UniqueName: \"kubernetes.io/projected/7dd70011-6085-49a8-b665-cb33157874fe-kube-api-access-pf557\") pod \"kas-connection-checker-685895c7c9-dj4nw\" (UID: \"7dd70011-6085-49a8-b665-cb33157874fe\") " pod="kube-system/kas-connection-checker-685895c7c9-dj4nw" Oct 30 21:42:32.689032 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.689012 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5qk\" (UniqueName: \"kubernetes.io/projected/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-kube-api-access-4r5qk\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:32.689117 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.689075 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g22b\" (UniqueName: \"kubernetes.io/projected/ef109382-58f3-45a9-bbd6-ef811dafc3d1-kube-api-access-5g22b\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:32.689980 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.689962 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-serving-cert\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:32.690036 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.690025 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:32.690093 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.690077 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0adf6cb3-60c7-471c-ba9e-62c236e4c7c4-config\") pod \"service-ca-operator-544d749c8c-brrm7\" (UID: \"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" Oct 30 21:42:32.726305 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726282 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.726420 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726360 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.726420 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726378 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726467 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726475 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726598 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726632 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726639 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726662 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726677 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726722 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726734 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:42:32.726758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726757 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726780 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726876 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726880 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726909 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726913 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726957 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.726973 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727023 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727160 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727184 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:42:32.727196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727197 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.727591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727223 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.727591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727230 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:42:32.727591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727237 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:42:32.727591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727263 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.727591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727237 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:42:32.727591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727214 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:42:32.727591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727547 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:42:32.727873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.727858 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:42:32.728243 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728226 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:42:32.728313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728227 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:42:32.728367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728354 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:42:32.728697 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728678 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6vmmq\"" Oct 30 21:42:32.728825 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728732 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:42:32.728825 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728791 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rfsrz\"" Oct 30 21:42:32.728825 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728805 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:42:32.728825 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728806 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:42:32.729023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728837 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:42:32.729023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728841 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:42:32.729023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728895 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tpd8h\"" Oct 30 21:42:32.729023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728899 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:42:32.729023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.728930 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fxktb\"" Oct 30 21:42:32.729256 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729238 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcz6k\"" Oct 30 21:42:32.729313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729248 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4c9l\"" Oct 30 21:42:32.729313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729262 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-spllf\"" Oct 30 21:42:32.729313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729275 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gbz5v\"" Oct 30 21:42:32.729313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729272 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5cjr4\"" Oct 30 21:42:32.729313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729302 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6j5v7\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729276 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qdnct\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729318 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-msl6j\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729374 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pvwl\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729383 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qtzdm\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729374 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hln5f\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729469 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729519 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729529 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xw9dh\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729532 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpjmc\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729528 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729535 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:42:32.729585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729542 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7m6w\" (UniqueName: \"kubernetes.io/projected/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-kube-api-access-w7m6w\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.729991 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.729971 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-fn8lc\"" Oct 30 21:42:32.730209 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.730193 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-konnectivity-ca\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:42:32.730745 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.730728 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-binary-copy\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.730812 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.730795 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.730870 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.730849 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9-agent-certs\") pod \"konnectivity-agent-v4lbg\" (UID: \"2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9\") " pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:42:32.730962 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.730945 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.731024 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.730977 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e1c3208-87b8-4a27-8538-6d0ddde14be7-data-volume\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.731024 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731003 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-var-lib-kubelet\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.731125 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731026 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/061b0a6d-64e8-4a45-8ada-59d74f6023a2-cert\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:32.731125 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731058 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.731125 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731066 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e1c3208-87b8-4a27-8538-6d0ddde14be7-data-volume\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.731125 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731090 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81526e36-032f-4f22-a94e-5e48680b7bc7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:32.731125 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731106 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-var-lib-kubelet\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731126 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-kubelet\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731160 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731189 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-systemd\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731218 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-multus-certs\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731242 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-script-lib\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731267 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731283 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/061b0a6d-64e8-4a45-8ada-59d74f6023a2-cert\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731281 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731291 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-dbus\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.731353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731311 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81526e36-032f-4f22-a94e-5e48680b7bc7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731362 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-systemd\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731416 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-conf\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731456 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-system-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731464 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731466 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731533 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-kubelet\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731564 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb00f28-52e5-48e2-a034-7193a74f8157-ovn-node-metrics-cert\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731602 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-script-lib\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731602 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-conf\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731607 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731679 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731705 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-kubelet-config\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731711 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb00f28-52e5-48e2-a034-7193a74f8157-ovn-node-metrics-cert\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.731748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731722 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-config\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731789 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6780389-4c8f-449a-a007-8682974061cb-tmp-dir\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731808 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhmd7\" (UniqueName: \"kubernetes.io/projected/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9-kube-api-access-lhmd7\") pod \"kas-connection-checker-685895c7c9-ptmmz\" (UID: \"f8ea0e11-f736-4db0-9ffe-d46ed7479dd9\") " pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731823 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvl8j\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731839 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mng\" (UniqueName: \"kubernetes.io/projected/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-kube-api-access-z7mng\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731859 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7z5\" (UniqueName: \"kubernetes.io/projected/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-api-access-8j7z5\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731881 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731896 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731892 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6780389-4c8f-449a-a007-8682974061cb-tmp-dir\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731913 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731914 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-config\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731935 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731972 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-log-socket\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732000 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.731994 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732045 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732089 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732125 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-k8s-cni-cncf-io\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732136 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysctl-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732149 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-var-lib-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732139 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732177 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-sys-fs\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732197 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kkzt\" (UniqueName: \"kubernetes.io/projected/061b0a6d-64e8-4a45-8ada-59d74f6023a2-kube-api-access-8kkzt\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732228 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732253 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732331 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732348 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cnibin\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.732367 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732386 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-host\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732418 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-cnibin\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732467 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732463 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75ql\" (UniqueName: \"kubernetes.io/projected/f0f22346-7f81-4d27-92b3-fde049d58d93-kube-api-access-m75ql\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732517 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-daemon-config\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732519 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-host\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732617 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6780389-4c8f-449a-a007-8682974061cb-hosts-file\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732669 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5n7q\" (UniqueName: \"kubernetes.io/projected/6a4f783d-df3d-4c74-939e-22bf246e9738-kube-api-access-w5n7q\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732693 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrzf\" (UniqueName: \"kubernetes.io/projected/c9a317df-0311-4ba6-bbb1-7556663b7bb5-kube-api-access-dvrzf\") pod \"network-check-source-6f78d59f99-7wr6f\" (UID: \"c9a317df-0311-4ba6-bbb1-7556663b7bb5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732737 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732761 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-sys\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732786 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p6rh\" (UniqueName: \"kubernetes.io/projected/3be119ed-6228-46e8-a71b-8f7fb6e812f8-kube-api-access-7p6rh\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732811 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-slash\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732833 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-node-log\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732811 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-daemon-config\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732878 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-sys\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732897 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-kubernetes\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732929 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732951 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-stats-auth\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732950 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-kubernetes\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.732983 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-netd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733004 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e1c3208-87b8-4a27-8538-6d0ddde14be7-crio-socket\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733022 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733061 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733131 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-registration-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733175 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjt2t\" (UniqueName: \"kubernetes.io/projected/df4f0be9-a14d-4120-a10f-c815830aef30-kube-api-access-kjt2t\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733199 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-stats-auth\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733220 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs4q\" (UniqueName: \"kubernetes.io/projected/1f607097-ece8-4833-8f34-361ac77ca2ee-kube-api-access-vcs4q\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733257 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-lib-modules\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733327 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-bin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733330 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5214b527-3150-40e4-9162-f712d602c83a-original-pull-secret\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733361 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-lib-modules\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733367 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f607097-ece8-4833-8f34-361ac77ca2ee-serving-cert\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733439 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-netns\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733468 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-systemd-units\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733524 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f607097-ece8-4833-8f34-361ac77ca2ee-serving-cert\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733540 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-service-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733577 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-systemd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733623 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-bin\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.733623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733645 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ce4f09-438e-4534-aeae-d427c599641a-config-volume\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733659 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-snapshots\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733659 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f607097-ece8-4833-8f34-361ac77ca2ee-service-ca-bundle\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733689 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733721 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733736 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-snapshots\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733746 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-modprobe-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733776 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6dfl\" (UniqueName: \"kubernetes.io/projected/7b5b5c38-04af-42ba-9845-41520e647cbe-kube-api-access-n6dfl\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733797 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f22346-7f81-4d27-92b3-fde049d58d93-serviceca\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733820 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ce4f09-438e-4534-aeae-d427c599641a-config-volume\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.733824 4933 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733824 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733833 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-modprobe-d\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733873 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-default-certificate\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.733888 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.233873489 +0000 UTC m=+1.915023381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.733979 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-multus\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734001 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-device-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734018 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysconfig\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734032 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f22346-7f81-4d27-92b3-fde049d58d93-serviceca\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734037 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f22346-7f81-4d27-92b3-fde049d58d93-host\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734079 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734092 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-etc-sysconfig\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734101 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-config\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734160 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734178 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-default-certificate\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734196 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cde9eb3a-08fa-49c1-bd82-38b414b0235f-host-slash\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734223 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734249 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc6h\" (UniqueName: \"kubernetes.io/projected/b6780389-4c8f-449a-a007-8682974061cb-kube-api-access-spc6h\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734299 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-ovnkube-config\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734306 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9v74\" (UniqueName: \"kubernetes.io/projected/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96-kube-api-access-v9v74\") pod \"migrator-66df5d496b-5b5xn\" (UID: \"fe67f1be-5081-45d9-b7a5-6fbfac3c5a96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734314 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734357 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734397 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734423 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734434 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734449 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-tmp\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734484 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734539 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-system-cni-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734556 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkg7\" (UniqueName: \"kubernetes.io/projected/8b8dc362-93da-466a-b4bd-7d33a25a00e4-kube-api-access-xmkg7\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734559 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f607097-ece8-4833-8f34-361ac77ca2ee-tmp\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734580 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7bz\" (UniqueName: \"kubernetes.io/projected/c0a1d58f-d414-42c1-8505-79acb178f8cf-kube-api-access-xc7bz\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734599 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-socket-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734606 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7-system-cni-dir\") pod \"multus-additional-cni-plugins-jw58x\" (UID: \"a1538ab8-ca5c-4f56-b86f-1a0ae2f313a7\") " pod="openshift-multus/multus-additional-cni-plugins-jw58x" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734616 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhq24\" (UniqueName: \"kubernetes.io/projected/3b72356e-8b7b-4392-859e-5ac980f759ef-kube-api-access-xhq24\") pod \"volume-data-source-validator-58c9f5f46d-85c8f\" (UID: \"3b72356e-8b7b-4392-859e-5ac980f759ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734639 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-run\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734660 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734673 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7b5b5c38-04af-42ba-9845-41520e647cbe-run\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734676 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-ovn\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734693 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvd5\" (UniqueName: \"kubernetes.io/projected/95ce4f09-438e-4534-aeae-d427c599641a-kube-api-access-zsvd5\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734734 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734786 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734829 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-trusted-ca\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734928 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-cnibin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734954 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-netns\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.734979 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cde9eb3a-08fa-49c1-bd82-38b414b0235f-iptables-alerter-script\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735008 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-hostroot\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735076 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gjp\" (UniqueName: \"kubernetes.io/projected/d0ee0cf5-2de4-493c-85fe-f97c527d5635-kube-api-access-54gjp\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735095 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-cni-binary-copy\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735113 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-etc-kubernetes\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735129 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735146 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735164 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735197 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-env-overrides\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735213 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cde9eb3a-08fa-49c1-bd82-38b414b0235f-iptables-alerter-script\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735215 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee0cf5-2de4-493c-85fe-f97c527d5635-trusted-ca\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735252 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735257 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735289 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjnd\" (UniqueName: \"kubernetes.io/projected/b10b2b7a-ecaf-434a-8248-0bdfd7df1667-kube-api-access-zrjnd\") pod \"kas-connection-checker-685895c7c9-tvk4h\" (UID: \"b10b2b7a-ecaf-434a-8248-0bdfd7df1667\") " pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735333 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4f783d-df3d-4c74-939e-22bf246e9738-cni-binary-copy\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.735320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735350 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb00f28-52e5-48e2-a034-7193a74f8157-env-overrides\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735411 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8dc362-93da-466a-b4bd-7d33a25a00e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735456 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735533 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ce4f09-438e-4534-aeae-d427c599641a-tmp-dir\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735560 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dtl\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735579 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ce4f09-438e-4534-aeae-d427c599641a-tmp-dir\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735581 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735624 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0ee0cf5-2de4-493c-85fe-f97c527d5635-serving-cert\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735632 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8b8dc362-93da-466a-b4bd-7d33a25a00e4-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735655 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-socket-dir-parent\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735681 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-etc-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735709 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br58m\" (UniqueName: \"kubernetes.io/projected/fdb00f28-52e5-48e2-a034-7193a74f8157-kube-api-access-br58m\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735735 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-conf-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735733 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735786 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0ee0cf5-2de4-493c-85fe-f97c527d5635-serving-cert\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735788 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.737339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.735823 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhd6\" (UniqueName: \"kubernetes.io/projected/cde9eb3a-08fa-49c1-bd82-38b414b0235f-kube-api-access-5bhd6\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:42:32.789151 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.789124 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhmd7\" (UniqueName: \"kubernetes.io/projected/f8ea0e11-f736-4db0-9ffe-d46ed7479dd9-kube-api-access-lhmd7\") pod \"kas-connection-checker-685895c7c9-ptmmz\" (UID: \"f8ea0e11-f736-4db0-9ffe-d46ed7479dd9\") " pod="kube-system/kas-connection-checker-685895c7c9-ptmmz" Oct 30 21:42:32.789289 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.789273 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvl8j\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.790122 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.790107 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7z5\" (UniqueName: \"kubernetes.io/projected/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-api-access-8j7z5\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.793575 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.793554 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kkzt\" (UniqueName: \"kubernetes.io/projected/061b0a6d-64e8-4a45-8ada-59d74f6023a2-kube-api-access-8kkzt\") pod \"ingress-canary-gmcqj\" (UID: \"061b0a6d-64e8-4a45-8ada-59d74f6023a2\") " pod="openshift-ingress-canary/ingress-canary-gmcqj" Oct 30 21:42:32.793789 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.793770 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5n7q\" (UniqueName: \"kubernetes.io/projected/6a4f783d-df3d-4c74-939e-22bf246e9738-kube-api-access-w5n7q\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.793945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.793929 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75ql\" (UniqueName: \"kubernetes.io/projected/f0f22346-7f81-4d27-92b3-fde049d58d93-kube-api-access-m75ql\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:42:32.794190 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.794172 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p6rh\" (UniqueName: \"kubernetes.io/projected/3be119ed-6228-46e8-a71b-8f7fb6e812f8-kube-api-access-7p6rh\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.794393 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.794377 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrzf\" (UniqueName: \"kubernetes.io/projected/c9a317df-0311-4ba6-bbb1-7556663b7bb5-kube-api-access-dvrzf\") pod \"network-check-source-6f78d59f99-7wr6f\" (UID: \"c9a317df-0311-4ba6-bbb1-7556663b7bb5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-7wr6f" Oct 30 21:42:32.794520 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.794481 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.794558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.794520 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6dfl\" (UniqueName: \"kubernetes.io/projected/7b5b5c38-04af-42ba-9845-41520e647cbe-kube-api-access-n6dfl\") pod \"tuned-ct72s\" (UID: \"7b5b5c38-04af-42ba-9845-41520e647cbe\") " pod="openshift-cluster-node-tuning-operator/tuned-ct72s" Oct 30 21:42:32.805138 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.805121 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhq24\" (UniqueName: \"kubernetes.io/projected/3b72356e-8b7b-4392-859e-5ac980f759ef-kube-api-access-xhq24\") pod \"volume-data-source-validator-58c9f5f46d-85c8f\" (UID: \"3b72356e-8b7b-4392-859e-5ac980f759ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-85c8f" Oct 30 21:42:32.805474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.805453 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mng\" (UniqueName: \"kubernetes.io/projected/f9ea4697-1a09-47e3-b6bc-ed44dae88ded-kube-api-access-z7mng\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn\" (UID: \"f9ea4697-1a09-47e3-b6bc-ed44dae88ded\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" Oct 30 21:42:32.805653 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.805615 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gjp\" (UniqueName: \"kubernetes.io/projected/d0ee0cf5-2de4-493c-85fe-f97c527d5635-kube-api-access-54gjp\") pod \"console-operator-7cd56b8dbc-q69rp\" (UID: \"d0ee0cf5-2de4-493c-85fe-f97c527d5635\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:32.806151 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.806135 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjt2t\" (UniqueName: \"kubernetes.io/projected/df4f0be9-a14d-4120-a10f-c815830aef30-kube-api-access-kjt2t\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.806253 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.806239 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7rp\" (UniqueName: \"kubernetes.io/projected/55e5bc9a-1f53-4a57-88a5-446df5df92f0-kube-api-access-8v7rp\") pod \"network-check-target-sz4m5\" (UID: \"55e5bc9a-1f53-4a57-88a5-446df5df92f0\") " pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:32.806289 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.806277 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br58m\" (UniqueName: \"kubernetes.io/projected/fdb00f28-52e5-48e2-a034-7193a74f8157-kube-api-access-br58m\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.807183 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.807167 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhd6\" (UniqueName: \"kubernetes.io/projected/cde9eb3a-08fa-49c1-bd82-38b414b0235f-kube-api-access-5bhd6\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:42:32.808805 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.808786 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dtl\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.808881 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.808821 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs4q\" (UniqueName: \"kubernetes.io/projected/1f607097-ece8-4833-8f34-361ac77ca2ee-kube-api-access-vcs4q\") pod \"insights-operator-7c4986d449-zxfcs\" (UID: \"1f607097-ece8-4833-8f34-361ac77ca2ee\") " pod="openshift-insights/insights-operator-7c4986d449-zxfcs" Oct 30 21:42:32.808881 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.808854 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.809078 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.809057 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjnd\" (UniqueName: \"kubernetes.io/projected/b10b2b7a-ecaf-434a-8248-0bdfd7df1667-kube-api-access-zrjnd\") pod \"kas-connection-checker-685895c7c9-tvk4h\" (UID: \"b10b2b7a-ecaf-434a-8248-0bdfd7df1667\") " pod="kube-system/kas-connection-checker-685895c7c9-tvk4h" Oct 30 21:42:32.809320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.809304 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvd5\" (UniqueName: \"kubernetes.io/projected/95ce4f09-438e-4534-aeae-d427c599641a-kube-api-access-zsvd5\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.809356 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.809311 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkg7\" (UniqueName: \"kubernetes.io/projected/8b8dc362-93da-466a-b4bd-7d33a25a00e4-kube-api-access-xmkg7\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:32.809465 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.809451 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9v74\" (UniqueName: \"kubernetes.io/projected/fe67f1be-5081-45d9-b7a5-6fbfac3c5a96-kube-api-access-v9v74\") pod \"migrator-66df5d496b-5b5xn\" (UID: \"fe67f1be-5081-45d9-b7a5-6fbfac3c5a96\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5b5xn" Oct 30 21:42:32.810352 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.810336 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7bz\" (UniqueName: \"kubernetes.io/projected/c0a1d58f-d414-42c1-8505-79acb178f8cf-kube-api-access-xc7bz\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:32.816457 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.816441 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc6h\" (UniqueName: \"kubernetes.io/projected/b6780389-4c8f-449a-a007-8682974061cb-kube-api-access-spc6h\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:42:32.836431 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836412 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:32.836528 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836434 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-ovn\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.836528 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836449 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-cnibin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.836528 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836463 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-netns\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.836528 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836477 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-hostroot\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.836528 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836511 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-etc-kubernetes\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836532 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-ovn\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836534 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836552 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-cnibin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836556 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-hostroot\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836572 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-netns\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.836561 4933 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836573 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836585 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-etc-kubernetes\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.836595 4933 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.836621 4933 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836650 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.836656 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.336625606 +0000 UTC m=+2.017775485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.836688 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.336675578 +0000 UTC m=+2.017825450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836620 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.836707 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.336698377 +0000 UTC m=+2.017848251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836733 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-socket-dir-parent\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.836748 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836753 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-etc-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836768 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-conf-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836782 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836784 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-socket-dir-parent\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836806 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-kubelet\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836816 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-etc-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836815 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-conf-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836847 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-multus-certs\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836865 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-kubelet\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836868 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.836887 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.336872862 +0000 UTC m=+2.018022753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836903 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-multus-certs\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836905 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-etc-selinux\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836922 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-dbus\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836938 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-system-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836952 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-kubelet\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836968 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.836987 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837005 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-system-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837024 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-kubelet\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837035 4933 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837046 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837054 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-kubelet-config\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837070 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-dbus\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837078 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.337069481 +0000 UTC m=+2.018219351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837091 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5214b527-3150-40e4-9162-f712d602c83a-kubelet-config\") pod \"global-pull-secret-syncer-qdn6l\" (UID: \"5214b527-3150-40e4-9162-f712d602c83a\") " pod="kube-system/global-pull-secret-syncer-qdn6l" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837119 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-log-socket\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837146 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837171 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-k8s-cni-cncf-io\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837193 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-var-lib-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837240 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837240 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-k8s-cni-cncf-io\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837248 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837298 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-var-lib-openvswitch\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837317 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.337302102 +0000 UTC m=+2.018451989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837325 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-log-socket\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837342 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-sys-fs\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837359 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837375 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6780389-4c8f-449a-a007-8682974061cb-hosts-file\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837391 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-slash\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837404 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-node-log\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837411 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-sys-fs\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837434 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-slash\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837452 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b6780389-4c8f-449a-a007-8682974061cb-hosts-file\") pod \"node-resolver-lq586\" (UID: \"b6780389-4c8f-449a-a007-8682974061cb\") " pod="openshift-dns/node-resolver-lq586" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837441 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-netd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837468 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837478 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-netd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837480 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:32.837483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837526 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e1c3208-87b8-4a27-8538-6d0ddde14be7-crio-socket\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837526 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-node-log\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837551 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.337541863 +0000 UTC m=+2.018691771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837481 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e1c3208-87b8-4a27-8538-6d0ddde14be7-crio-socket\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837572 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837590 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-registration-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837610 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-bin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837626 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-netns\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837631 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-run-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837639 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-systemd-units\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837668 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-run-netns\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837675 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-systemd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837684 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-registration-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837691 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-bin\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837671 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-systemd-units\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837704 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-bin\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837708 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-run-systemd\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837745 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837755 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-cni-bin\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837771 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-multus\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837793 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-device-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837817 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f22346-7f81-4d27-92b3-fde049d58d93-host\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837828 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-host-var-lib-cni-multus\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837840 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837857 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f22346-7f81-4d27-92b3-fde049d58d93-host\") pod \"node-ca-wqw28\" (UID: \"f0f22346-7f81-4d27-92b3-fde049d58d93\") " pod="openshift-image-registry/node-ca-wqw28" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837869 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cde9eb3a-08fa-49c1-bd82-38b414b0235f-host-slash\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837877 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-device-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837891 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4f783d-df3d-4c74-939e-22bf246e9738-multus-cni-dir\") pod \"multus-vvgps\" (UID: \"6a4f783d-df3d-4c74-939e-22bf246e9738\") " pod="openshift-multus/multus-vvgps" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837897 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837922 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837927 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cde9eb3a-08fa-49c1-bd82-38b414b0235f-host-slash\") pod \"iptables-alerter-59qkx\" (UID: \"cde9eb3a-08fa-49c1-bd82-38b414b0235f\") " pod="openshift-network-operator/iptables-alerter-59qkx" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837947 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837977 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-socket-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.837989 4933 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.837993 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb00f28-52e5-48e2-a034-7193a74f8157-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vgt94\" (UID: \"fdb00f28-52e5-48e2-a034-7193a74f8157\") " pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.838028 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.338015428 +0000 UTC m=+2.019165314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.838065 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f0be9-a14d-4120-a10f-c815830aef30-socket-dir\") pod \"aws-ebs-csi-driver-node-xp69k\" (UID: \"df4f0be9-a14d-4120-a10f-c815830aef30\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xp69k" Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.838104 4933 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:32.838141 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:42:33.338128761 +0000 UTC m=+2.019278636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : secret "metrics-daemon-secret" not found Oct 30 21:42:32.839137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.838189 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e1c3208-87b8-4a27-8538-6d0ddde14be7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:32.875282 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:32.875260 4933 scope.go:122] "RemoveContainer" containerID="a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4" Oct 30 21:42:33.240658 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.240600 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:33.240768 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.240738 4933 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:33.240803 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.240788 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.240774882 +0000 UTC m=+2.921924753 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:33.340881 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.340862 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:33.340964 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.340892 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:33.340964 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.340908 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:33.340964 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.340932 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.340966 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.340995 4933 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341028 4933 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341036 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341025452 +0000 UTC m=+3.022175324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341035 4933 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.340997 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341053 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341064 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341076 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341054721 +0000 UTC m=+3.022204613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341101 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341092753 +0000 UTC m=+3.022242625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341103 4933 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:33.341120 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341120 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341113882 +0000 UTC m=+3.022263752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341145 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341128749 +0000 UTC m=+3.022278625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341158 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341151749 +0000 UTC m=+3.022301619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.341180 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.341225 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.341242 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341287 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341296 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341316 4933 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341316 4933 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341327 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341312344 +0000 UTC m=+3.022462232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341350 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341339138 +0000 UTC m=+3.022489023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:33.341482 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.341363 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.341357086 +0000 UTC m=+3.022506961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : secret "metrics-daemon-secret" not found Oct 30 21:42:33.518401 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.518355 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:42:33.518800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.518786 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/1.log" Oct 30 21:42:33.518860 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.518816 4933 generic.go:356] "Generic (PLEG): container finished" podID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" containerID="7635807adab89df622766c75e56216813a814cd9e52e4e6fb88eac0b374c4e61" exitCode=255 Oct 30 21:42:33.518903 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.518866 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" event={"ID":"d0ee0cf5-2de4-493c-85fe-f97c527d5635","Type":"ContainerDied","Data":"7635807adab89df622766c75e56216813a814cd9e52e4e6fb88eac0b374c4e61"} Oct 30 21:42:33.518956 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.518910 4933 scope.go:122] "RemoveContainer" containerID="a1de5e20d8667d81977b722c084f2beeb29a71c5505e7dd11c5ae37a4cde2ce4" Oct 30 21:42:33.519357 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:33.519340 4933 scope.go:122] "RemoveContainer" containerID="7635807adab89df622766c75e56216813a814cd9e52e4e6fb88eac0b374c4e61" Oct 30 21:42:33.519573 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:33.519548 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q69rp_openshift-console-operator(d0ee0cf5-2de4-493c-85fe-f97c527d5635)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" podUID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" Oct 30 21:42:34.249882 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.249851 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:34.250034 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.249984 4933 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:42:34.250077 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.250040 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls podName:ef109382-58f3-45a9-bbd6-ef811dafc3d1 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.250024409 +0000 UTC m=+4.931174282 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ncxvk" (UID: "ef109382-58f3-45a9-bbd6-ef811dafc3d1") : secret "samples-operator-tls" not found Oct 30 21:42:34.350715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.350687 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:34.350715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.350719 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350834 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350837 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350848 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350856 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.350860 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.350893 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350909 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.350889339 +0000 UTC m=+5.032039212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:34.350932 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350933 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.350924386 +0000 UTC m=+5.032074263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350960 4933 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.350978 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.350991 4933 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351000 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.35098449 +0000 UTC m=+5.032134375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : secret "router-metrics-certs-default" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351043 4933 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351054 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs podName:c0a1d58f-d414-42c1-8505-79acb178f8cf nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.351040092 +0000 UTC m=+5.032189964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs") pod "network-metrics-daemon-g7hlg" (UID: "c0a1d58f-d414-42c1-8505-79acb178f8cf") : secret "metrics-daemon-secret" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351082 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.35107034 +0000 UTC m=+5.032220235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.351100 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.351131 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.351171 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351181 4933 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351212 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.351202091 +0000 UTC m=+5.032351989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.351231 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351253 4933 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351264 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.351253236 +0000 UTC m=+5.032403111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351281 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.351273324 +0000 UTC m=+5.032423230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351309 4933 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:34.351346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:34.351338 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:36.351330535 +0000 UTC m=+5.032480451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:34.523304 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.523235 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:42:34.615661 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:34.615640 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:42:35.315187 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:35.315155 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:35.315481 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:35.315467 4933 scope.go:122] "RemoveContainer" containerID="7635807adab89df622766c75e56216813a814cd9e52e4e6fb88eac0b374c4e61" Oct 30 21:42:35.315688 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:35.315665 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q69rp_openshift-console-operator(d0ee0cf5-2de4-493c-85fe-f97c527d5635)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" podUID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" Oct 30 21:42:35.851572 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:35.851549 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lq586_b6780389-4c8f-449a-a007-8682974061cb/dns-node-resolver/0.log" Oct 30 21:42:35.907684 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:35.907665 4933 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:35.907956 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:35.907943 4933 scope.go:122] "RemoveContainer" containerID="7635807adab89df622766c75e56216813a814cd9e52e4e6fb88eac0b374c4e61" Oct 30 21:42:35.908097 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:35.908081 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-q69rp_openshift-console-operator(d0ee0cf5-2de4-493c-85fe-f97c527d5635)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" podUID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" Oct 30 21:42:36.266978 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.266925 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:36.267266 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.267248 4933 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:42:36.269550 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.269532 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef109382-58f3-45a9-bbd6-ef811dafc3d1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ncxvk\" (UID: \"ef109382-58f3-45a9-bbd6-ef811dafc3d1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:36.367595 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367575 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:36.367698 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367606 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:36.367698 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367625 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:36.367698 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367670 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:36.367698 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367689 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367713 4933 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367739 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367757 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64975494cd-d2ggk: secret "image-registry-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367763 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:40.367750802 +0000 UTC m=+9.048900708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367765 4933 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367715 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367739 4933 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367805 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls podName:04ed16df-9be5-40ee-b222-ed7c2d11455a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:40.367789901 +0000 UTC m=+9.048939795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls") pod "image-registry-64975494cd-d2ggk" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a") : secret "image-registry-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367809 4933 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b86c7dc-q72nj: secret "image-registry-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367836 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:40.36782553 +0000 UTC m=+9.048975407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.367854 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls podName:64bf86a4-9e40-472e-89a5-83e099fd7953 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:40.367846013 +0000 UTC m=+9.048995901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls") pod "image-registry-675b86c7dc-q72nj" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953") : secret "image-registry-tls" not found Oct 30 21:42:36.367912 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367886 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:36.368405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367919 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:36.368405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.367960 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:36.368405 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.368033 4933 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:36.368405 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.368053 4933 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:36.368405 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.368060 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:40.368051205 +0000 UTC m=+9.049201075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:36.368405 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.368078 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle podName:3be119ed-6228-46e8-a71b-8f7fb6e812f8 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:40.368066286 +0000 UTC m=+9.049216179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle") pod "router-default-7444c59677-nrb6q" (UID: "3be119ed-6228-46e8-a71b-8f7fb6e812f8") : configmap references non-existent config key: service-ca.crt Oct 30 21:42:36.368405 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:36.368111 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:40.368094436 +0000 UTC m=+9.049244343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:36.370026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.370003 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1d58f-d414-42c1-8505-79acb178f8cf-metrics-certs\") pod \"network-metrics-daemon-g7hlg\" (UID: \"c0a1d58f-d414-42c1-8505-79acb178f8cf\") " pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:36.370120 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.370059 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be119ed-6228-46e8-a71b-8f7fb6e812f8-metrics-certs\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:36.479783 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.479762 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" Oct 30 21:42:36.484077 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.484060 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g7hlg" Oct 30 21:42:36.852858 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.852793 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk"] Oct 30 21:42:36.897354 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.897329 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g7hlg"] Oct 30 21:42:36.904200 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:36.904156 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a1d58f_d414_42c1_8505_79acb178f8cf.slice/crio-acbf8e50125e1ec07941debf41d2be858be8d461c3fb53ded77a212a02201068 WatchSource:0}: Error finding container acbf8e50125e1ec07941debf41d2be858be8d461c3fb53ded77a212a02201068: Status 404 returned error can't find the container with id acbf8e50125e1ec07941debf41d2be858be8d461c3fb53ded77a212a02201068 Oct 30 21:42:36.906097 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:36.906082 4933 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:42:37.052075 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:37.052050 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wqw28_f0f22346-7f81-4d27-92b3-fde049d58d93/node-ca/0.log" Oct 30 21:42:37.538800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:37.538774 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7hlg" event={"ID":"c0a1d58f-d414-42c1-8505-79acb178f8cf","Type":"ContainerStarted","Data":"acbf8e50125e1ec07941debf41d2be858be8d461c3fb53ded77a212a02201068"} Oct 30 21:42:37.539859 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:37.539837 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" event={"ID":"ef109382-58f3-45a9-bbd6-ef811dafc3d1","Type":"ContainerStarted","Data":"e3f89b97aac6021e9cfad38839d7668a1223c431d9450165f5b8396e46cc92ca"} Oct 30 21:42:37.651874 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:37.651848 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gmcqj_061b0a6d-64e8-4a45-8ada-59d74f6023a2/serve-healthcheck-canary/0.log" Oct 30 21:42:37.733985 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:37.733964 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:38.052873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:38.052846 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5b5xn_fe67f1be-5081-45d9-b7a5-6fbfac3c5a96/migrator/0.log" Oct 30 21:42:38.453004 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:38.452980 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn_f9ea4697-1a09-47e3-b6bc-ed44dae88ded/kube-storage-version-migrator-operator/0.log" Oct 30 21:42:38.545323 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:38.545296 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7hlg" event={"ID":"c0a1d58f-d414-42c1-8505-79acb178f8cf","Type":"ContainerStarted","Data":"cbd1675d9b40a2ac8eb6e9113b34bcfb6aae3f796f2b442725de34881ca6b92d"} Oct 30 21:42:38.545426 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:38.545352 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g7hlg" event={"ID":"c0a1d58f-d414-42c1-8505-79acb178f8cf","Type":"ContainerStarted","Data":"bddd6f296b2192882c08d603a9ade68e204aec6def51384077532c64a1d1d620"} Oct 30 21:42:39.550290 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:39.550255 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" event={"ID":"ef109382-58f3-45a9-bbd6-ef811dafc3d1","Type":"ContainerStarted","Data":"01b90fe0ed8ef8ce664e3658f65e3ff68c0794407c6516917e79329cd266c586"} Oct 30 21:42:39.550652 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:39.550307 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ncxvk" event={"ID":"ef109382-58f3-45a9-bbd6-ef811dafc3d1","Type":"ContainerStarted","Data":"460e6c485c4c127dc54043e09e1a117a2730692f4f9ebbaf0d3618bd0540a1ef"} Oct 30 21:42:40.403321 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403283 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403333 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403358 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403389 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403428 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403430 4933 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403510 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls podName:8b8dc362-93da-466a-b4bd-7d33a25a00e4 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:48.403478414 +0000 UTC m=+17.084628285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-ljcqn" (UID: "8b8dc362-93da-466a-b4bd-7d33a25a00e4") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403430 4933 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:42:40.403531 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403530 4933 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:42:40.403928 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403541 4933 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:42:40.403928 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403578 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:40.403928 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403600 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls podName:95ce4f09-438e-4534-aeae-d427c599641a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:48.403580625 +0000 UTC m=+17.084730497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls") pod "dns-default-pb7j7" (UID: "95ce4f09-438e-4534-aeae-d427c599641a") : secret "dns-default-metrics-tls" not found Oct 30 21:42:40.403928 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403629 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls podName:5e1c3208-87b8-4a27-8538-6d0ddde14be7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:48.403619959 +0000 UTC m=+17.084769836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m2qts" (UID: "5e1c3208-87b8-4a27-8538-6d0ddde14be7") : secret "insights-runtime-extractor-tls" not found Oct 30 21:42:40.403928 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:40.403668 4933 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert podName:81526e36-032f-4f22-a94e-5e48680b7bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:48.403641761 +0000 UTC m=+17.084791643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-5xpsq" (UID: "81526e36-032f-4f22-a94e-5e48680b7bc7") : secret "networking-console-plugin-cert" not found Oct 30 21:42:40.403928 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403714 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:40.404201 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.403999 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3be119ed-6228-46e8-a71b-8f7fb6e812f8-service-ca-bundle\") pod \"router-default-7444c59677-nrb6q\" (UID: \"3be119ed-6228-46e8-a71b-8f7fb6e812f8\") " pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:40.405964 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.405941 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"image-registry-675b86c7dc-q72nj\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:40.406041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.405979 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"image-registry-64975494cd-d2ggk\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:40.679938 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.679882 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:40.680971 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.680954 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:40.685587 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.685567 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:40.869874 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.869844 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:40.870124 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:40.870114 4933 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:41.159184 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.159155 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7444c59677-nrb6q"] Oct 30 21:42:41.162137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.162108 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64975494cd-d2ggk"] Oct 30 21:42:41.162237 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:41.162174 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be119ed_6228_46e8_a71b_8f7fb6e812f8.slice/crio-928c23c31f08fb60bd0abcbf382cfe7ad9dedb50dc4077d6ad815995df387cdf WatchSource:0}: Error finding container 928c23c31f08fb60bd0abcbf382cfe7ad9dedb50dc4077d6ad815995df387cdf: Status 404 returned error can't find the container with id 928c23c31f08fb60bd0abcbf382cfe7ad9dedb50dc4077d6ad815995df387cdf Oct 30 21:42:41.165224 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:41.165207 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ed16df_9be5_40ee_b222_ed7c2d11455a.slice/crio-62fe3b8f8c8a9745159080d29bda42c55138c145e64ba1affc229f80c03277a6 WatchSource:0}: Error finding container 62fe3b8f8c8a9745159080d29bda42c55138c145e64ba1affc229f80c03277a6: Status 404 returned error can't find the container with id 62fe3b8f8c8a9745159080d29bda42c55138c145e64ba1affc229f80c03277a6 Oct 30 21:42:41.208999 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.208929 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675b86c7dc-q72nj"] Oct 30 21:42:41.211575 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:41.211554 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bf86a4_9e40_472e_89a5_83e099fd7953.slice/crio-8537741a5904a6568c50a079e567f0dcf443c86fe3292c902c9d35f82637eef7 WatchSource:0}: Error finding container 8537741a5904a6568c50a079e567f0dcf443c86fe3292c902c9d35f82637eef7: Status 404 returned error can't find the container with id 8537741a5904a6568c50a079e567f0dcf443c86fe3292c902c9d35f82637eef7 Oct 30 21:42:41.558517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.558454 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" event={"ID":"64bf86a4-9e40-472e-89a5-83e099fd7953","Type":"ContainerStarted","Data":"e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac"} Oct 30 21:42:41.558688 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.558543 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" event={"ID":"64bf86a4-9e40-472e-89a5-83e099fd7953","Type":"ContainerStarted","Data":"8537741a5904a6568c50a079e567f0dcf443c86fe3292c902c9d35f82637eef7"} Oct 30 21:42:41.560658 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.560619 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7444c59677-nrb6q" event={"ID":"3be119ed-6228-46e8-a71b-8f7fb6e812f8","Type":"ContainerStarted","Data":"d659f6aefb24bd90f57de7c7eea562b778ed80b0559cc4a1bf5c275a900e1c8f"} Oct 30 21:42:41.560797 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.560665 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7444c59677-nrb6q" event={"ID":"3be119ed-6228-46e8-a71b-8f7fb6e812f8","Type":"ContainerStarted","Data":"928c23c31f08fb60bd0abcbf382cfe7ad9dedb50dc4077d6ad815995df387cdf"} Oct 30 21:42:41.562279 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.562249 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" event={"ID":"04ed16df-9be5-40ee-b222-ed7c2d11455a","Type":"ContainerStarted","Data":"7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e"} Oct 30 21:42:41.562381 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.562283 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" event={"ID":"04ed16df-9be5-40ee-b222-ed7c2d11455a","Type":"ContainerStarted","Data":"62fe3b8f8c8a9745159080d29bda42c55138c145e64ba1affc229f80c03277a6"} Oct 30 21:42:41.680632 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.680603 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:41.683065 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.683042 4933 patch_prober.go:29] interesting pod/router-default-7444c59677-nrb6q container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Oct 30 21:42:41.683065 ip-10-0-131-250 kubenswrapper[4933]: [+]has-synced ok Oct 30 21:42:41.683065 ip-10-0-131-250 kubenswrapper[4933]: [+]process-running ok Oct 30 21:42:41.683065 ip-10-0-131-250 kubenswrapper[4933]: healthz check failed Oct 30 21:42:41.683278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:41.683110 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-7444c59677-nrb6q" podUID="3be119ed-6228-46e8-a71b-8f7fb6e812f8" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Oct 30 21:42:42.241159 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:42.241135 4933 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-250.us-west-1.compute.internal" event="NodeReady" Oct 30 21:42:42.565695 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:42.565665 4933 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:42.565695 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:42.565669 4933 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:42.681577 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:42.681558 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:44.571856 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:44.571824 4933 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:45.458647 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:45.458617 4933 scope.go:122] "RemoveContainer" containerID="7635807adab89df622766c75e56216813a814cd9e52e4e6fb88eac0b374c4e61" Oct 30 21:42:45.577076 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:45.577051 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:42:45.577391 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:45.577105 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" event={"ID":"d0ee0cf5-2de4-493c-85fe-f97c527d5635","Type":"ContainerStarted","Data":"df04703745108f1ae18db388edcd61f7b724c95d7eb414c690bcbcffc5434ba5"} Oct 30 21:42:45.577391 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:45.577332 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:45.578165 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:45.578142 4933 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-q69rp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body= Oct 30 21:42:45.578258 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:45.578184 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" podUID="d0ee0cf5-2de4-493c-85fe-f97c527d5635" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" Oct 30 21:42:45.619418 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:45.619398 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:42:46.585749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:46.585723 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-q69rp" Oct 30 21:42:47.668640 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:47.668614 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:47.669074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:47.668702 4933 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:42:47.669922 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:47.669901 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7444c59677-nrb6q" Oct 30 21:42:47.900630 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:47.900600 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64975494cd-d2ggk"] Oct 30 21:42:48.474437 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.474408 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:48.474597 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.474464 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:48.474597 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.474508 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:48.474597 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.474535 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:48.477077 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.477047 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ce4f09-438e-4534-aeae-d427c599641a-metrics-tls\") pod \"dns-default-pb7j7\" (UID: \"95ce4f09-438e-4534-aeae-d427c599641a\") " pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:48.477077 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.477057 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/81526e36-032f-4f22-a94e-5e48680b7bc7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-5xpsq\" (UID: \"81526e36-032f-4f22-a94e-5e48680b7bc7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:48.477077 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.477057 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8b8dc362-93da-466a-b4bd-7d33a25a00e4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-ljcqn\" (UID: \"8b8dc362-93da-466a-b4bd-7d33a25a00e4\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:48.477249 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.477115 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e1c3208-87b8-4a27-8538-6d0ddde14be7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m2qts\" (UID: \"5e1c3208-87b8-4a27-8538-6d0ddde14be7\") " pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:48.480011 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.479994 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" Oct 30 21:42:48.486721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.486700 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:48.487797 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.487777 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m2qts" Oct 30 21:42:48.774579 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.774481 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" Oct 30 21:42:48.849001 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.848976 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-5xpsq"] Oct 30 21:42:48.891075 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:48.891052 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81526e36_032f_4f22_a94e_5e48680b7bc7.slice/crio-01c2ea8ba4c3d67622878e22140d29e8552b313821257932ed4dd2e585d71b88 WatchSource:0}: Error finding container 01c2ea8ba4c3d67622878e22140d29e8552b313821257932ed4dd2e585d71b88: Status 404 returned error can't find the container with id 01c2ea8ba4c3d67622878e22140d29e8552b313821257932ed4dd2e585d71b88 Oct 30 21:42:48.900216 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.900139 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pb7j7"] Oct 30 21:42:48.902046 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:48.902024 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ce4f09_438e_4534_aeae_d427c599641a.slice/crio-8a2391ac27056aacdaa59dfe0f091de0ae5aece1b7126ca84db0b7a40f3d147c WatchSource:0}: Error finding container 8a2391ac27056aacdaa59dfe0f091de0ae5aece1b7126ca84db0b7a40f3d147c: Status 404 returned error can't find the container with id 8a2391ac27056aacdaa59dfe0f091de0ae5aece1b7126ca84db0b7a40f3d147c Oct 30 21:42:48.928579 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:48.928558 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m2qts"] Oct 30 21:42:48.940170 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:48.940150 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1c3208_87b8_4a27_8538_6d0ddde14be7.slice/crio-8e1f2ba113d88a645d42c32740559d98a4eacec6de0d8652457a008fff29117d WatchSource:0}: Error finding container 8e1f2ba113d88a645d42c32740559d98a4eacec6de0d8652457a008fff29117d: Status 404 returned error can't find the container with id 8e1f2ba113d88a645d42c32740559d98a4eacec6de0d8652457a008fff29117d Oct 30 21:42:49.143363 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.143342 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn"] Oct 30 21:42:49.145808 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:42:49.145783 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8dc362_93da_466a_b4bd_7d33a25a00e4.slice/crio-e44d3e78eebdf555f15413e534781673b4eea815abc887fa5abe9b8daa420eea WatchSource:0}: Error finding container e44d3e78eebdf555f15413e534781673b4eea815abc887fa5abe9b8daa420eea: Status 404 returned error can't find the container with id e44d3e78eebdf555f15413e534781673b4eea815abc887fa5abe9b8daa420eea Oct 30 21:42:49.536573 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.536544 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vgt94" Oct 30 21:42:49.592093 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.592064 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pb7j7" event={"ID":"95ce4f09-438e-4534-aeae-d427c599641a","Type":"ContainerStarted","Data":"8a2391ac27056aacdaa59dfe0f091de0ae5aece1b7126ca84db0b7a40f3d147c"} Oct 30 21:42:49.593543 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.593515 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qts" event={"ID":"5e1c3208-87b8-4a27-8538-6d0ddde14be7","Type":"ContainerStarted","Data":"9f6e41f17566526cb9c7a4b7bf37566f70dd7531a917a15a4ac623d27cb68f8a"} Oct 30 21:42:49.593678 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.593569 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qts" event={"ID":"5e1c3208-87b8-4a27-8538-6d0ddde14be7","Type":"ContainerStarted","Data":"8e1f2ba113d88a645d42c32740559d98a4eacec6de0d8652457a008fff29117d"} Oct 30 21:42:49.594669 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.594651 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" event={"ID":"81526e36-032f-4f22-a94e-5e48680b7bc7","Type":"ContainerStarted","Data":"01c2ea8ba4c3d67622878e22140d29e8552b313821257932ed4dd2e585d71b88"} Oct 30 21:42:49.595786 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.595767 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" event={"ID":"8b8dc362-93da-466a-b4bd-7d33a25a00e4","Type":"ContainerStarted","Data":"e44d3e78eebdf555f15413e534781673b4eea815abc887fa5abe9b8daa420eea"} Oct 30 21:42:49.690970 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:49.690930 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:42:50.602586 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:50.602553 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qts" event={"ID":"5e1c3208-87b8-4a27-8538-6d0ddde14be7","Type":"ContainerStarted","Data":"174513ce26aef1b5ae778275a3c2f61d292024d222f8a1eebb80608e65477491"} Oct 30 21:42:51.608869 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:51.608789 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pb7j7" event={"ID":"95ce4f09-438e-4534-aeae-d427c599641a","Type":"ContainerStarted","Data":"f47e854f7bfd0bcffd7caa57a4238b232778ed2dc4de96d55c91302c839c5f65"} Oct 30 21:42:51.608869 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:51.608849 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pb7j7" event={"ID":"95ce4f09-438e-4534-aeae-d427c599641a","Type":"ContainerStarted","Data":"439509c9ee6914dc577cabc66d0289dd03671d3a7217adff5f9adc16a6058afa"} Oct 30 21:42:51.609292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:51.609003 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pb7j7" Oct 30 21:42:51.610542 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:51.610512 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-5xpsq" event={"ID":"81526e36-032f-4f22-a94e-5e48680b7bc7","Type":"ContainerStarted","Data":"b2486737d9be84906d5c2f7b82e3a5767abdda6a28588e4dd74402f2ef860735"} Oct 30 21:42:51.612025 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:51.611998 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-ljcqn" event={"ID":"8b8dc362-93da-466a-b4bd-7d33a25a00e4","Type":"ContainerStarted","Data":"0f0977255467d56cd51bcba0a36382bae8bc2791b04eb70cca588a428a90f421"} Oct 30 21:42:52.617296 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:52.617258 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m2qts" event={"ID":"5e1c3208-87b8-4a27-8538-6d0ddde14be7","Type":"ContainerStarted","Data":"9f7b77aa5d7855ee852048f746df7255d5d8651eefc0a5c9cb71e2f436c28665"} Oct 30 21:42:57.286754 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:57.286719 4933 patch_prober.go:29] interesting pod/image-registry-675b86c7dc-q72nj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 21:42:57.287260 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:57.286769 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" podUID="64bf86a4-9e40-472e-89a5-83e099fd7953" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:57.905927 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:57.905900 4933 patch_prober.go:29] interesting pod/image-registry-64975494cd-d2ggk container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 21:42:57.906079 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:57.905945 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" podUID="04ed16df-9be5-40ee-b222-ed7c2d11455a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:42:58.542250 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:42:58.542214 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:42:59.099713 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:59.099685 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v4lbg" Oct 30 21:42:59.695406 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:59.695377 4933 patch_prober.go:29] interesting pod/image-registry-675b86c7dc-q72nj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 21:42:59.695790 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:42:59.695424 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" podUID="64bf86a4-9e40-472e-89a5-83e099fd7953" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:43:01.619634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:01.619609 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pb7j7" Oct 30 21:43:04.619283 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:04.619257 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sz4m5" Oct 30 21:43:04.911761 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:04.911688 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z4m48"] Oct 30 21:43:04.947461 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:04.947437 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.010224 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.010191 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:43:05.010348 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.010227 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xc8xg\"" Oct 30 21:43:05.010348 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.010233 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:43:05.089079 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.089054 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:43:05.089210 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.089199 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:43:05.091545 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091525 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-sys\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091631 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091552 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-root\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091631 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091584 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg77t\" (UniqueName: \"kubernetes.io/projected/b2392a57-633c-42a4-92e5-64fbb91ce9fc-kube-api-access-hg77t\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091723 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091664 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-textfile\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091757 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091737 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-tls\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091792 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091774 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091835 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091805 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2392a57-633c-42a4-92e5-64fbb91ce9fc-metrics-client-ca\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091870 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091843 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.091916 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.091876 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-wtmp\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192470 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192413 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-wtmp\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192470 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192454 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-sys\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192471 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-root\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192529 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg77t\" (UniqueName: \"kubernetes.io/projected/b2392a57-633c-42a4-92e5-64fbb91ce9fc-kube-api-access-hg77t\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192546 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-textfile\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192554 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-sys\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192563 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-tls\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192591 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-wtmp\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192614 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192949 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192656 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2392a57-633c-42a4-92e5-64fbb91ce9fc-metrics-client-ca\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192949 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192662 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2392a57-633c-42a4-92e5-64fbb91ce9fc-root\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.192949 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.192674 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.193090 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.193020 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-textfile\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.193333 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.193311 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.193390 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.193323 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2392a57-633c-42a4-92e5-64fbb91ce9fc-metrics-client-ca\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.195101 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.195081 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-tls\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.195164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.195081 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2392a57-633c-42a4-92e5-64fbb91ce9fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.249580 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.249561 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg77t\" (UniqueName: \"kubernetes.io/projected/b2392a57-633c-42a4-92e5-64fbb91ce9fc-kube-api-access-hg77t\") pod \"node-exporter-z4m48\" (UID: \"b2392a57-633c-42a4-92e5-64fbb91ce9fc\") " pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.300209 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.300188 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z4m48" Oct 30 21:43:05.331852 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:43:05.331827 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2392a57_633c_42a4_92e5_64fbb91ce9fc.slice/crio-379316ef21b447530b7e7aa7e6cdc35321e1fd5a01fd31f3545c2d37c975b8e2 WatchSource:0}: Error finding container 379316ef21b447530b7e7aa7e6cdc35321e1fd5a01fd31f3545c2d37c975b8e2: Status 404 returned error can't find the container with id 379316ef21b447530b7e7aa7e6cdc35321e1fd5a01fd31f3545c2d37c975b8e2 Oct 30 21:43:05.663155 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:05.663123 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4m48" event={"ID":"b2392a57-633c-42a4-92e5-64fbb91ce9fc","Type":"ContainerStarted","Data":"379316ef21b447530b7e7aa7e6cdc35321e1fd5a01fd31f3545c2d37c975b8e2"} Oct 30 21:43:07.672077 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:07.672042 4933 generic.go:356] "Generic (PLEG): container finished" podID="b2392a57-633c-42a4-92e5-64fbb91ce9fc" containerID="c5145c3e5d5248e584d9a54134bc163a32222fe1f7976bad9eb7e46e7d16297c" exitCode=0 Oct 30 21:43:07.672539 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:07.672095 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4m48" event={"ID":"b2392a57-633c-42a4-92e5-64fbb91ce9fc","Type":"ContainerDied","Data":"c5145c3e5d5248e584d9a54134bc163a32222fe1f7976bad9eb7e46e7d16297c"} Oct 30 21:43:07.905697 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:07.905673 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:43:08.681508 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:08.681448 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4m48" event={"ID":"b2392a57-633c-42a4-92e5-64fbb91ce9fc","Type":"ContainerStarted","Data":"c05b22fa32ee3c09395918d85e33f78f87e4dce5a73dc211a8082767a426d94b"} Oct 30 21:43:08.681882 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:08.681565 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z4m48" event={"ID":"b2392a57-633c-42a4-92e5-64fbb91ce9fc","Type":"ContainerStarted","Data":"27b3dbf9f2df3c6b68f7719a8dab375885f3a43489c43a167bcea1fd81cd7ee8"} Oct 30 21:43:08.800968 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:08.800917 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z4m48" podStartSLOduration=2.786396668 podStartE2EDuration="4.800905365s" totalImagesPullingTime="1.919979733s" totalInitContainerRuntime="94.528964ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:08.800129432 +0000 UTC m=+37.481279338" watchObservedRunningTime="2026-10-30 21:43:08.800905365 +0000 UTC m=+37.482055250" Oct 30 21:43:09.695999 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:09.695973 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:43:12.923679 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:12.923552 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" podUID="04ed16df-9be5-40ee-b222-ed7c2d11455a" containerName="registry" containerID="cri-o://7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e" gracePeriod=30 Oct 30 21:43:13.159538 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.159516 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:43:13.254610 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254552 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.254610 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254593 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.254763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254640 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.254763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254658 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.254763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254674 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.254763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254693 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.254763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254743 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.254995 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.254770 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\") pod \"04ed16df-9be5-40ee-b222-ed7c2d11455a\" (UID: \"04ed16df-9be5-40ee-b222-ed7c2d11455a\") " Oct 30 21:43:13.255466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.255134 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:13.255466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.255424 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:13.257296 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.257250 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:13.257519 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.257473 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "kube-api-access-dvl8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:13.257641 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.257514 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:13.257641 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.257567 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:13.257759 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.257708 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:13.261967 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.261938 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted" pod "04ed16df-9be5-40ee-b222-ed7c2d11455a" (UID: "04ed16df-9be5-40ee-b222-ed7c2d11455a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:13.355784 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355759 4933 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-bound-sa-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.355784 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355781 4933 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-certificates\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.355969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355792 4933 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-image-registry-private-configuration\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.355969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355801 4933 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-registry-tls\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.355969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355810 4933 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/04ed16df-9be5-40ee-b222-ed7c2d11455a-trusted-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.355969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355818 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvl8j\" (UniqueName: \"kubernetes.io/projected/04ed16df-9be5-40ee-b222-ed7c2d11455a-kube-api-access-dvl8j\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.355969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355828 4933 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/04ed16df-9be5-40ee-b222-ed7c2d11455a-installation-pull-secrets\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.355969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.355837 4933 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/04ed16df-9be5-40ee-b222-ed7c2d11455a-ca-trust-extracted\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:13.700151 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.700120 4933 generic.go:356] "Generic (PLEG): container finished" podID="04ed16df-9be5-40ee-b222-ed7c2d11455a" containerID="7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e" exitCode=0 Oct 30 21:43:13.700284 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.700167 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" Oct 30 21:43:13.700284 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.700206 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" event={"ID":"04ed16df-9be5-40ee-b222-ed7c2d11455a","Type":"ContainerDied","Data":"7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e"} Oct 30 21:43:13.700284 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.700259 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64975494cd-d2ggk" event={"ID":"04ed16df-9be5-40ee-b222-ed7c2d11455a","Type":"ContainerDied","Data":"62fe3b8f8c8a9745159080d29bda42c55138c145e64ba1affc229f80c03277a6"} Oct 30 21:43:13.700284 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.700279 4933 scope.go:122] "RemoveContainer" containerID="7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e" Oct 30 21:43:13.709295 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.709271 4933 scope.go:122] "RemoveContainer" containerID="7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e" Oct 30 21:43:13.709626 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:43:13.709604 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e\": container with ID starting with 7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e not found: ID does not exist" containerID="7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e" Oct 30 21:43:13.709716 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.709636 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e"} err="failed to get container status \"7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e\": rpc error: code = NotFound desc = could not find container \"7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e\": container with ID starting with 7153fa6fd28dc04e2d6c4442ff8a772262affe2a45151af48c1a5552fac3162e not found: ID does not exist" Oct 30 21:43:13.869011 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.868986 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64975494cd-d2ggk"] Oct 30 21:43:13.919289 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:13.919269 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64975494cd-d2ggk"] Oct 30 21:43:15.464160 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:15.464126 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ed16df-9be5-40ee-b222-ed7c2d11455a" path="/var/lib/kubelet/pods/04ed16df-9be5-40ee-b222-ed7c2d11455a/volumes" Oct 30 21:43:20.277877 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:20.277842 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675b86c7dc-q72nj"] Oct 30 21:43:28.542692 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:43:28.542654 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:43:41.805814 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:41.805774 4933 generic.go:356] "Generic (PLEG): container finished" podID="f9ea4697-1a09-47e3-b6bc-ed44dae88ded" containerID="a384ea4a56d8fed954a87c76b0c5a0846b99e65a5dcf8f77c0019d8c78227fbd" exitCode=0 Oct 30 21:43:41.806182 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:41.805838 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" event={"ID":"f9ea4697-1a09-47e3-b6bc-ed44dae88ded","Type":"ContainerDied","Data":"a384ea4a56d8fed954a87c76b0c5a0846b99e65a5dcf8f77c0019d8c78227fbd"} Oct 30 21:43:41.806182 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:41.806125 4933 scope.go:122] "RemoveContainer" containerID="a384ea4a56d8fed954a87c76b0c5a0846b99e65a5dcf8f77c0019d8c78227fbd" Oct 30 21:43:42.812366 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:42.812325 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wp4zn" event={"ID":"f9ea4697-1a09-47e3-b6bc-ed44dae88ded","Type":"ContainerStarted","Data":"49d12a52bfe4d8f4c04499f353e3aeeceb339c1056237d9e3b40d52ed1535f10"} Oct 30 21:43:45.298432 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.298392 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" podUID="64bf86a4-9e40-472e-89a5-83e099fd7953" containerName="registry" containerID="cri-o://e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac" gracePeriod=30 Oct 30 21:43:45.544731 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.544707 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:43:45.574759 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574703 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.574759 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574739 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.574945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574776 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.574945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574795 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.574945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574814 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.574945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574831 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.574945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574868 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.574945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.574898 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\") pod \"64bf86a4-9e40-472e-89a5-83e099fd7953\" (UID: \"64bf86a4-9e40-472e-89a5-83e099fd7953\") " Oct 30 21:43:45.575278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.575162 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:45.575391 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.575293 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:43:45.577292 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.577268 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:45.577799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.577772 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:43:45.577937 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.577911 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "kube-api-access-27dtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:45.577994 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.577963 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:45.577994 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.577967 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:43:45.583572 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.583548 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted" pod "64bf86a4-9e40-472e-89a5-83e099fd7953" (UID: "64bf86a4-9e40-472e-89a5-83e099fd7953"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:43:45.675360 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675339 4933 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-installation-pull-secrets\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.675360 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675359 4933 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/64bf86a4-9e40-472e-89a5-83e099fd7953-image-registry-private-configuration\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.675473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675370 4933 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-certificates\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.675473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675379 4933 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-registry-tls\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.675473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675387 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27dtl\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-kube-api-access-27dtl\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.675473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675395 4933 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64bf86a4-9e40-472e-89a5-83e099fd7953-bound-sa-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.675473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675403 4933 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/64bf86a4-9e40-472e-89a5-83e099fd7953-ca-trust-extracted\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.675473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.675411 4933 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/64bf86a4-9e40-472e-89a5-83e099fd7953-trusted-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:43:45.823017 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.822984 4933 generic.go:356] "Generic (PLEG): container finished" podID="64bf86a4-9e40-472e-89a5-83e099fd7953" containerID="e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac" exitCode=0 Oct 30 21:43:45.823130 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.823021 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" event={"ID":"64bf86a4-9e40-472e-89a5-83e099fd7953","Type":"ContainerDied","Data":"e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac"} Oct 30 21:43:45.823130 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.823074 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" event={"ID":"64bf86a4-9e40-472e-89a5-83e099fd7953","Type":"ContainerDied","Data":"8537741a5904a6568c50a079e567f0dcf443c86fe3292c902c9d35f82637eef7"} Oct 30 21:43:45.823130 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.823093 4933 scope.go:122] "RemoveContainer" containerID="e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac" Oct 30 21:43:45.823130 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.823033 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b86c7dc-q72nj" Oct 30 21:43:45.831648 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.831632 4933 scope.go:122] "RemoveContainer" containerID="e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac" Oct 30 21:43:45.833833 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:43:45.832588 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac\": container with ID starting with e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac not found: ID does not exist" containerID="e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac" Oct 30 21:43:45.833833 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.832628 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac"} err="failed to get container status \"e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac\": rpc error: code = NotFound desc = could not find container \"e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac\": container with ID starting with e1f55efecbc88fdf50276a56ed1ee0fcbc46d26ab3cbd9f87ff48834659a10ac not found: ID does not exist" Oct 30 21:43:45.994576 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.994551 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675b86c7dc-q72nj"] Oct 30 21:43:45.999371 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:45.999352 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-675b86c7dc-q72nj"] Oct 30 21:43:47.463259 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:47.463224 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bf86a4-9e40-472e-89a5-83e099fd7953" path="/var/lib/kubelet/pods/64bf86a4-9e40-472e-89a5-83e099fd7953/volumes" Oct 30 21:43:56.859698 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:56.859664 4933 generic.go:356] "Generic (PLEG): container finished" podID="0adf6cb3-60c7-471c-ba9e-62c236e4c7c4" containerID="4084af60217f3f36b5f437d529628dfe27f705697bc50d5021fd4a77d7412890" exitCode=0 Oct 30 21:43:56.860059 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:56.859732 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" event={"ID":"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4","Type":"ContainerDied","Data":"4084af60217f3f36b5f437d529628dfe27f705697bc50d5021fd4a77d7412890"} Oct 30 21:43:56.860059 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:56.860028 4933 scope.go:122] "RemoveContainer" containerID="4084af60217f3f36b5f437d529628dfe27f705697bc50d5021fd4a77d7412890" Oct 30 21:43:57.865201 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:43:57.865173 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-brrm7" event={"ID":"0adf6cb3-60c7-471c-ba9e-62c236e4c7c4","Type":"ContainerStarted","Data":"de7d10d50bf345e7ae5d8c5034f97f42201a3014497e6aee58a73b20a68f9d61"} Oct 30 21:43:58.542841 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:43:58.542810 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:44:01.879358 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:44:01.879318 4933 generic.go:356] "Generic (PLEG): container finished" podID="1f607097-ece8-4833-8f34-361ac77ca2ee" containerID="a1a0cf21b42f4cf0d631f4d7adc864591f42fd43200d33ebfb203d0c911d9704" exitCode=0 Oct 30 21:44:01.879812 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:44:01.879392 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zxfcs" event={"ID":"1f607097-ece8-4833-8f34-361ac77ca2ee","Type":"ContainerDied","Data":"a1a0cf21b42f4cf0d631f4d7adc864591f42fd43200d33ebfb203d0c911d9704"} Oct 30 21:44:01.879812 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:44:01.879678 4933 scope.go:122] "RemoveContainer" containerID="a1a0cf21b42f4cf0d631f4d7adc864591f42fd43200d33ebfb203d0c911d9704" Oct 30 21:44:02.884574 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:44:02.884540 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zxfcs" event={"ID":"1f607097-ece8-4833-8f34-361ac77ca2ee","Type":"ContainerStarted","Data":"39f80630a5aee6984f2d42e6f6398198192411fdabea55c182c10cc84c055e96"} Oct 30 21:44:28.541470 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:44:28.541438 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:44:58.542395 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:44:58.542319 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:45:28.542346 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:45:28.542310 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:45:58.542229 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:45:58.542192 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:46:28.542592 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:46:28.542481 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:46:37.183479 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.183448 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f"] Oct 30 21:46:37.183901 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.183732 4933 cpu_manager.go:414] "containerMap: removing container" podUID="64bf86a4-9e40-472e-89a5-83e099fd7953" containerName="registry" Oct 30 21:46:37.183901 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.183743 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bf86a4-9e40-472e-89a5-83e099fd7953" containerName="registry" Oct 30 21:46:37.183901 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.183756 4933 cpu_manager.go:414] "containerMap: removing container" podUID="04ed16df-9be5-40ee-b222-ed7c2d11455a" containerName="registry" Oct 30 21:46:37.183901 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.183761 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ed16df-9be5-40ee-b222-ed7c2d11455a" containerName="registry" Oct 30 21:46:37.183901 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.183823 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ed16df-9be5-40ee-b222-ed7c2d11455a" containerName="registry" Oct 30 21:46:37.183901 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.183835 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bf86a4-9e40-472e-89a5-83e099fd7953" containerName="registry" Oct 30 21:46:37.186928 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.186906 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.290695 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.290663 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:46:37.290836 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.290713 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:46:37.290836 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.290830 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jlqzr\"" Oct 30 21:46:37.291772 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.291754 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f"] Oct 30 21:46:37.321449 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.321430 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.321559 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.321464 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7pjv\" (UniqueName: \"kubernetes.io/projected/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-kube-api-access-v7pjv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.321559 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.321513 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.422084 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.422059 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7pjv\" (UniqueName: \"kubernetes.io/projected/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-kube-api-access-v7pjv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.422215 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.422096 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.422215 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.422131 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.422425 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.422410 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.422508 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.422457 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.481377 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.481321 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7pjv\" (UniqueName: \"kubernetes.io/projected/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-kube-api-access-v7pjv\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.501579 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.501556 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:46:37.809325 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.809297 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq"] Oct 30 21:46:37.812813 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.812798 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:37.866700 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.866681 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-p4gpj\"" Oct 30 21:46:37.870765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.870740 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq"] Oct 30 21:46:37.901986 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.901964 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z"] Oct 30 21:46:37.905188 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.905175 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:37.926250 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.926222 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tmq\" (UniqueName: \"kubernetes.io/projected/fbc94cef-a2b1-48ad-8478-d2c880e9251a-kube-api-access-g8tmq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:37.926346 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.926266 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:37.926385 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.926358 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:37.931383 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.931363 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f"] Oct 30 21:46:37.953860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:37.953839 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb7207d_d19b_45d7_9a90_24f429eaa6b7.slice/crio-e507a127df77772851cee7da70a4b2729eabdc7fceb3bb44aca7a0d99fb22916 WatchSource:0}: Error finding container e507a127df77772851cee7da70a4b2729eabdc7fceb3bb44aca7a0d99fb22916: Status 404 returned error can't find the container with id e507a127df77772851cee7da70a4b2729eabdc7fceb3bb44aca7a0d99fb22916 Oct 30 21:46:37.966005 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.965983 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-cqfmc\"" Oct 30 21:46:37.975049 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.975031 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc"] Oct 30 21:46:37.978316 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.978303 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:37.982912 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:37.982887 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z"] Oct 30 21:46:38.027270 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027239 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgnh\" (UniqueName: \"kubernetes.io/projected/0d2e75f0-1f35-4323-abf7-d22f0c436455-kube-api-access-4tgnh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.027364 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027277 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:38.027364 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027300 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srsnw\" (UniqueName: \"kubernetes.io/projected/22cc790b-d980-4c54-9f80-4c2238c05054-kube-api-access-srsnw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.027521 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027479 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.027561 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027544 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tmq\" (UniqueName: \"kubernetes.io/projected/fbc94cef-a2b1-48ad-8478-d2c880e9251a-kube-api-access-g8tmq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:38.027606 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027577 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:38.027657 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027623 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:38.027657 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027633 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.027756 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027661 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.027829 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027810 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.027887 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.027835 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:38.037244 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.037217 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc"] Oct 30 21:46:38.067520 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.067458 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-66kdj\"" Oct 30 21:46:38.086831 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.086800 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tmq\" (UniqueName: \"kubernetes.io/projected/fbc94cef-a2b1-48ad-8478-d2c880e9251a-kube-api-access-g8tmq\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:38.087293 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.087277 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks"] Oct 30 21:46:38.090626 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.090613 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.128643 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.128620 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:46:38.128809 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.128793 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.128859 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.128829 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhf2k\" (UniqueName: \"kubernetes.io/projected/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-kube-api-access-vhf2k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.128906 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.128863 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgnh\" (UniqueName: \"kubernetes.io/projected/0d2e75f0-1f35-4323-abf7-d22f0c436455-kube-api-access-4tgnh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.128953 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.128935 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srsnw\" (UniqueName: \"kubernetes.io/projected/22cc790b-d980-4c54-9f80-4c2238c05054-kube-api-access-srsnw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.128995 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.128966 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.128995 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.128992 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.129075 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.129010 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.129110 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.129096 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.129152 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.129112 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.129187 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.129163 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.129319 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.129303 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.129386 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.129372 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.129422 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.129407 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.147865 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.147844 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks"] Oct 30 21:46:38.168521 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.168481 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-92lj2\"" Oct 30 21:46:38.188172 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.188149 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgnh\" (UniqueName: \"kubernetes.io/projected/0d2e75f0-1f35-4323-abf7-d22f0c436455-kube-api-access-4tgnh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.188578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.188168 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srsnw\" (UniqueName: \"kubernetes.io/projected/22cc790b-d980-4c54-9f80-4c2238c05054-kube-api-access-srsnw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.219680 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.219650 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:46:38.229711 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.229692 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhf2k\" (UniqueName: \"kubernetes.io/projected/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-kube-api-access-vhf2k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.229756 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.229732 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.229803 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.229757 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.230044 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.230027 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.230102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.230054 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.287359 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.287333 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhf2k\" (UniqueName: \"kubernetes.io/projected/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-kube-api-access-vhf2k\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.300220 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.300200 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:46:38.408048 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.408015 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:46:38.422278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.422244 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" event={"ID":"9fb7207d-d19b-45d7-9a90-24f429eaa6b7","Type":"ContainerStarted","Data":"e507a127df77772851cee7da70a4b2729eabdc7fceb3bb44aca7a0d99fb22916"} Oct 30 21:46:38.496449 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.496426 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq"] Oct 30 21:46:38.498100 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:38.498072 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc94cef_a2b1_48ad_8478_d2c880e9251a.slice/crio-0cb688922a80972dc95b7c81c8a388207f2c527e77ac65986aaab223e16eeb22 WatchSource:0}: Error finding container 0cb688922a80972dc95b7c81c8a388207f2c527e77ac65986aaab223e16eeb22: Status 404 returned error can't find the container with id 0cb688922a80972dc95b7c81c8a388207f2c527e77ac65986aaab223e16eeb22 Oct 30 21:46:38.587809 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.587783 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z"] Oct 30 21:46:38.589532 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:38.589508 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2e75f0_1f35_4323_abf7_d22f0c436455.slice/crio-821a4cba0182748e3c0a8faa5b91d1bd8e8de8b6aa5dec5090ac2afef054cffd WatchSource:0}: Error finding container 821a4cba0182748e3c0a8faa5b91d1bd8e8de8b6aa5dec5090ac2afef054cffd: Status 404 returned error can't find the container with id 821a4cba0182748e3c0a8faa5b91d1bd8e8de8b6aa5dec5090ac2afef054cffd Oct 30 21:46:38.665962 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.665940 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc"] Oct 30 21:46:38.668087 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:38.668061 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cc790b_d980_4c54_9f80_4c2238c05054.slice/crio-cda5f3da2208edbab0c7c21b82c3f74e6a24b6280114ced57f716775ec4e4be0 WatchSource:0}: Error finding container cda5f3da2208edbab0c7c21b82c3f74e6a24b6280114ced57f716775ec4e4be0: Status 404 returned error can't find the container with id cda5f3da2208edbab0c7c21b82c3f74e6a24b6280114ced57f716775ec4e4be0 Oct 30 21:46:38.775464 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:38.775444 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks"] Oct 30 21:46:38.779287 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:38.779259 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c87fc4f_6bb4_4cc1_a76b_fa457438619b.slice/crio-28c4fc4edf8cdaa68a88545b6766f258182e9dba4442afa61d85ca882f278a9a WatchSource:0}: Error finding container 28c4fc4edf8cdaa68a88545b6766f258182e9dba4442afa61d85ca882f278a9a: Status 404 returned error can't find the container with id 28c4fc4edf8cdaa68a88545b6766f258182e9dba4442afa61d85ca882f278a9a Oct 30 21:46:39.428510 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:39.428447 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" event={"ID":"1c87fc4f-6bb4-4cc1-a76b-fa457438619b","Type":"ContainerStarted","Data":"28c4fc4edf8cdaa68a88545b6766f258182e9dba4442afa61d85ca882f278a9a"} Oct 30 21:46:39.430023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:39.429994 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" event={"ID":"fbc94cef-a2b1-48ad-8478-d2c880e9251a","Type":"ContainerStarted","Data":"0cb688922a80972dc95b7c81c8a388207f2c527e77ac65986aaab223e16eeb22"} Oct 30 21:46:39.431665 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:39.431641 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" event={"ID":"22cc790b-d980-4c54-9f80-4c2238c05054","Type":"ContainerStarted","Data":"cda5f3da2208edbab0c7c21b82c3f74e6a24b6280114ced57f716775ec4e4be0"} Oct 30 21:46:39.433098 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:39.433074 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" event={"ID":"0d2e75f0-1f35-4323-abf7-d22f0c436455","Type":"ContainerStarted","Data":"821a4cba0182748e3c0a8faa5b91d1bd8e8de8b6aa5dec5090ac2afef054cffd"} Oct 30 21:46:40.472097 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.472068 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk"] Oct 30 21:46:40.476656 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.476632 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.553791 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.553747 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.553969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.553795 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.553969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.553830 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrk6f\" (UniqueName: \"kubernetes.io/projected/b58b60e2-ea09-4c93-8ede-b11251b32719-kube-api-access-qrk6f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.583140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.583094 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-s24cn\"" Oct 30 21:46:40.584589 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.584555 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk"] Oct 30 21:46:40.656221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.655053 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrk6f\" (UniqueName: \"kubernetes.io/projected/b58b60e2-ea09-4c93-8ede-b11251b32719-kube-api-access-qrk6f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.656221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.655188 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.656221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.655223 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.656221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.655675 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.656221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.656111 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.694763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.692889 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25"] Oct 30 21:46:40.697537 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.696897 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.724731 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.724627 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrk6f\" (UniqueName: \"kubernetes.io/projected/b58b60e2-ea09-4c93-8ede-b11251b32719-kube-api-access-qrk6f\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.756074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.756043 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26s6r\" (UniqueName: \"kubernetes.io/projected/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-kube-api-access-26s6r\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.757262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.756094 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.757262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.756116 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.759859 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.759835 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25"] Oct 30 21:46:40.760180 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.760161 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2zlmm\"" Oct 30 21:46:40.798361 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.798280 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:46:40.856792 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.856766 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26s6r\" (UniqueName: \"kubernetes.io/projected/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-kube-api-access-26s6r\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.856914 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.856816 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.856914 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.856838 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.857184 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.857165 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.857616 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.857597 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:40.938918 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:40.937106 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26s6r\" (UniqueName: \"kubernetes.io/projected/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-kube-api-access-26s6r\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:41.018334 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:41.018007 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:46:41.227974 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:41.227937 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk"] Oct 30 21:46:41.235213 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:41.235166 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58b60e2_ea09_4c93_8ede_b11251b32719.slice/crio-61252a5e089c1743e09a1e3cbd3a8527b40cb0da358eff64695925de2b3d636d WatchSource:0}: Error finding container 61252a5e089c1743e09a1e3cbd3a8527b40cb0da358eff64695925de2b3d636d: Status 404 returned error can't find the container with id 61252a5e089c1743e09a1e3cbd3a8527b40cb0da358eff64695925de2b3d636d Oct 30 21:46:41.416869 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:41.416838 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25"] Oct 30 21:46:41.426871 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:41.426839 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9901e1bd_c6a6_4c45_8480_8cb2fe1a0e4c.slice/crio-c3ab26d4017e1736e791c039622c7e1b968d6ddf70ccc01de442083563bef85c WatchSource:0}: Error finding container c3ab26d4017e1736e791c039622c7e1b968d6ddf70ccc01de442083563bef85c: Status 404 returned error can't find the container with id c3ab26d4017e1736e791c039622c7e1b968d6ddf70ccc01de442083563bef85c Oct 30 21:46:41.448311 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:41.448249 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" event={"ID":"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c","Type":"ContainerStarted","Data":"c3ab26d4017e1736e791c039622c7e1b968d6ddf70ccc01de442083563bef85c"} Oct 30 21:46:41.449960 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:41.449894 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" event={"ID":"b58b60e2-ea09-4c93-8ede-b11251b32719","Type":"ContainerStarted","Data":"61252a5e089c1743e09a1e3cbd3a8527b40cb0da358eff64695925de2b3d636d"} Oct 30 21:46:42.441661 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.440438 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq"] Oct 30 21:46:42.444457 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.443806 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.497687 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.496919 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:46:42.508822 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.508785 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v2z7t\"" Oct 30 21:46:42.551205 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.549634 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq"] Oct 30 21:46:42.574891 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.574860 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.574906 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.574938 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.574966 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.574994 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnzz\" (UniqueName: \"kubernetes.io/projected/acef2b25-5c28-44db-8c2b-ba32a12ddce6-kube-api-access-rfnzz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575255 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.575051 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575255 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.575073 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575255 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.575123 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.575255 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.575147 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.675839 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnzz\" (UniqueName: \"kubernetes.io/projected/acef2b25-5c28-44db-8c2b-ba32a12ddce6-kube-api-access-rfnzz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.675915 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.675934 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.675972 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.675994 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.676034 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.676052 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.676081 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.676102 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.676787 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.677307 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.677567 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.678015 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.678736 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.678293 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.694005 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.693943 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.700320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.700270 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.737871 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.737801 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnzz\" (UniqueName: \"kubernetes.io/projected/acef2b25-5c28-44db-8c2b-ba32a12ddce6-kube-api-access-rfnzz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.739200 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.739128 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acef2b25-5c28-44db-8c2b-ba32a12ddce6-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq\" (UID: \"acef2b25-5c28-44db-8c2b-ba32a12ddce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.751540 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.750113 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k"] Oct 30 21:46:42.757023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.756285 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.761720 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.761505 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:42.833003 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.828451 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k"] Oct 30 21:46:42.833003 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.828709 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-72426\"" Oct 30 21:46:42.880225 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.878025 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7a612dc-b817-4e16-90c3-a73530490ad9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.880225 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.878086 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.880225 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.878119 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.882083 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.880925 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.882083 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.880991 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.882083 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.881038 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwqx\" (UniqueName: \"kubernetes.io/projected/e7a612dc-b817-4e16-90c3-a73530490ad9-kube-api-access-lzwqx\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.883609 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.883577 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.883722 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.883686 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.883788 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.883772 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.984759 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.984797 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.984814 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwqx\" (UniqueName: \"kubernetes.io/projected/e7a612dc-b817-4e16-90c3-a73530490ad9-kube-api-access-lzwqx\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985133 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985299 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985540 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985668 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985711 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985785 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7a612dc-b817-4e16-90c3-a73530490ad9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985825 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.985854 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.986019 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.986906 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.986615 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.987702 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.987618 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7a612dc-b817-4e16-90c3-a73530490ad9-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.992149 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.992106 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:42.999283 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:42.999254 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:43.046529 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:43.046458 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7a612dc-b817-4e16-90c3-a73530490ad9-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:43.055426 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:43.055372 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwqx\" (UniqueName: \"kubernetes.io/projected/e7a612dc-b817-4e16-90c3-a73530490ad9-kube-api-access-lzwqx\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k\" (UID: \"e7a612dc-b817-4e16-90c3-a73530490ad9\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:43.075967 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:43.075710 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:43.254125 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:43.253874 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq"] Oct 30 21:46:43.262038 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:46:43.261998 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacef2b25_5c28_44db_8c2b_ba32a12ddce6.slice/crio-da075aa65d33e3720667166342a9bd03d1f111e0595776277947a4e570e43200 WatchSource:0}: Error finding container da075aa65d33e3720667166342a9bd03d1f111e0595776277947a4e570e43200: Status 404 returned error can't find the container with id da075aa65d33e3720667166342a9bd03d1f111e0595776277947a4e570e43200 Oct 30 21:46:43.472947 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:43.472830 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" event={"ID":"acef2b25-5c28-44db-8c2b-ba32a12ddce6","Type":"ContainerStarted","Data":"da075aa65d33e3720667166342a9bd03d1f111e0595776277947a4e570e43200"} Oct 30 21:46:43.526361 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:43.525888 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k"] Oct 30 21:46:44.484268 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:44.484178 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" event={"ID":"e7a612dc-b817-4e16-90c3-a73530490ad9","Type":"ContainerStarted","Data":"5b27f5f30376c9fca8af177e2fd782db3f41ffbb4db732010048348719a2431b"} Oct 30 21:46:53.831838 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:53.831795 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:53.832197 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:53.831890 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:53.832197 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:53.831937 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:53.848664 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:53.848572 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:53.848765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:53.848718 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:53.848828 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:53.848763 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:46:54.556701 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.556662 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" event={"ID":"acef2b25-5c28-44db-8c2b-ba32a12ddce6","Type":"ContainerStarted","Data":"93bbb939a0c8ea9833e5b3afc567cec2fe76a7dbeaaa15ae52dfc6338a8d128e"} Oct 30 21:46:54.558621 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.558583 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" event={"ID":"0d2e75f0-1f35-4323-abf7-d22f0c436455","Type":"ContainerStarted","Data":"a3eb22b9efffd38214a5300f6ec1727ab6ead5f45232113809b5bc5426e97063"} Oct 30 21:46:54.560756 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.560729 4933 generic.go:356] "Generic (PLEG): container finished" podID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerID="09b8ae9708532f3901fc0e5e7e9a26955623a7058ee3cfa1e29b6ed2a2e54b41" exitCode=0 Oct 30 21:46:54.560884 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.560812 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" event={"ID":"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c","Type":"ContainerDied","Data":"09b8ae9708532f3901fc0e5e7e9a26955623a7058ee3cfa1e29b6ed2a2e54b41"} Oct 30 21:46:54.563811 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.563790 4933 generic.go:356] "Generic (PLEG): container finished" podID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerID="2a371e753d8824540aa0f4901f8882aa544c92bc5ae32bf2fb1ab9f244533d23" exitCode=0 Oct 30 21:46:54.563906 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.563853 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" event={"ID":"9fb7207d-d19b-45d7-9a90-24f429eaa6b7","Type":"ContainerDied","Data":"2a371e753d8824540aa0f4901f8882aa544c92bc5ae32bf2fb1ab9f244533d23"} Oct 30 21:46:54.565614 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.565566 4933 generic.go:356] "Generic (PLEG): container finished" podID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerID="1384f910b419a6b5c8bf784efe6a12eeee12794a753b285b7bfac5629a346ca1" exitCode=0 Oct 30 21:46:54.565783 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.565765 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" event={"ID":"fbc94cef-a2b1-48ad-8478-d2c880e9251a","Type":"ContainerDied","Data":"1384f910b419a6b5c8bf784efe6a12eeee12794a753b285b7bfac5629a346ca1"} Oct 30 21:46:54.567990 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.567963 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" event={"ID":"e7a612dc-b817-4e16-90c3-a73530490ad9","Type":"ContainerStarted","Data":"f47e8d92fcfd4babccf53301e203497d5fe5dba67dcfa51b1b0705e8404397ba"} Oct 30 21:46:54.762330 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.762305 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:54.767689 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.767636 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podStartSLOduration=2.188532904 podStartE2EDuration="12.767619921s" totalImagesPullingTime="10.579087017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:54.731086886 +0000 UTC m=+263.412236781" watchObservedRunningTime="2026-10-30 21:46:54.767619921 +0000 UTC m=+263.448769815" Oct 30 21:46:54.767801 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.767716 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:54.973092 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:54.973034 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" podStartSLOduration=2.6780683339999998 podStartE2EDuration="12.973019942s" totalImagesPullingTime="10.294951608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:54.972208118 +0000 UTC m=+263.653358010" watchObservedRunningTime="2026-10-30 21:46:54.973019942 +0000 UTC m=+263.654169836" Oct 30 21:46:55.076162 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.076131 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:55.081852 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.081828 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:55.573074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.573044 4933 generic.go:356] "Generic (PLEG): container finished" podID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerID="1567b42c2f8c006be60f814c86bca8a150ce03d16cf55350d10191b3cd926416" exitCode=0 Oct 30 21:46:55.573242 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.573136 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" event={"ID":"1c87fc4f-6bb4-4cc1-a76b-fa457438619b","Type":"ContainerDied","Data":"1567b42c2f8c006be60f814c86bca8a150ce03d16cf55350d10191b3cd926416"} Oct 30 21:46:55.574682 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.574658 4933 generic.go:356] "Generic (PLEG): container finished" podID="22cc790b-d980-4c54-9f80-4c2238c05054" containerID="c11dc1336ffcaa395cd8bda0d3797c974d20b132266b8137d0b286e948bfaa8d" exitCode=0 Oct 30 21:46:55.574802 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.574758 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" event={"ID":"22cc790b-d980-4c54-9f80-4c2238c05054","Type":"ContainerDied","Data":"c11dc1336ffcaa395cd8bda0d3797c974d20b132266b8137d0b286e948bfaa8d"} Oct 30 21:46:55.576269 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.576245 4933 generic.go:356] "Generic (PLEG): container finished" podID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerID="a3eb22b9efffd38214a5300f6ec1727ab6ead5f45232113809b5bc5426e97063" exitCode=0 Oct 30 21:46:55.576365 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.576282 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" event={"ID":"0d2e75f0-1f35-4323-abf7-d22f0c436455","Type":"ContainerDied","Data":"a3eb22b9efffd38214a5300f6ec1727ab6ead5f45232113809b5bc5426e97063"} Oct 30 21:46:55.576535 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.576521 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:55.576779 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.576761 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:55.577697 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.577679 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:46:55.577822 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:55.577811 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k" Oct 30 21:46:56.583925 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:56.583885 4933 generic.go:356] "Generic (PLEG): container finished" podID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerID="d59d7a8f836c416d1dc500fd767660b89222683e481f1461110c3aa9c38e3dd6" exitCode=0 Oct 30 21:46:56.584452 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:56.583976 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" event={"ID":"b58b60e2-ea09-4c93-8ede-b11251b32719","Type":"ContainerDied","Data":"d59d7a8f836c416d1dc500fd767660b89222683e481f1461110c3aa9c38e3dd6"} Oct 30 21:46:56.587116 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:56.586973 4933 generic.go:356] "Generic (PLEG): container finished" podID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerID="e6132b0ec6fe4e03c3fae2d235eafb6decceb84ddee3b1f975feb0c2fd0fc7d1" exitCode=0 Oct 30 21:46:56.587116 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:56.587072 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" event={"ID":"9fb7207d-d19b-45d7-9a90-24f429eaa6b7","Type":"ContainerDied","Data":"e6132b0ec6fe4e03c3fae2d235eafb6decceb84ddee3b1f975feb0c2fd0fc7d1"} Oct 30 21:46:56.589451 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:56.589431 4933 generic.go:356] "Generic (PLEG): container finished" podID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerID="c70cbb61266e7ab832a42cd607480a51ee88208196a8a0e9db47d8978e86bb52" exitCode=0 Oct 30 21:46:56.589572 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:56.589532 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" event={"ID":"fbc94cef-a2b1-48ad-8478-d2c880e9251a","Type":"ContainerDied","Data":"c70cbb61266e7ab832a42cd607480a51ee88208196a8a0e9db47d8978e86bb52"} Oct 30 21:46:57.596923 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:57.596890 4933 generic.go:356] "Generic (PLEG): container finished" podID="22cc790b-d980-4c54-9f80-4c2238c05054" containerID="1954865128e84b71318e2fc87523d30984131de982843bcdd9427e6d991a322e" exitCode=0 Oct 30 21:46:57.597321 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:57.596967 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" event={"ID":"22cc790b-d980-4c54-9f80-4c2238c05054","Type":"ContainerDied","Data":"1954865128e84b71318e2fc87523d30984131de982843bcdd9427e6d991a322e"} Oct 30 21:46:57.599806 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:57.599777 4933 generic.go:356] "Generic (PLEG): container finished" podID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerID="49afab0bae93e3c1a28079523f1a3e20d73db36b4bbcb614b36bafbbe0e1c44d" exitCode=0 Oct 30 21:46:57.599921 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:57.599880 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" event={"ID":"0d2e75f0-1f35-4323-abf7-d22f0c436455","Type":"ContainerDied","Data":"49afab0bae93e3c1a28079523f1a3e20d73db36b4bbcb614b36bafbbe0e1c44d"} Oct 30 21:46:57.601996 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:57.601965 4933 generic.go:356] "Generic (PLEG): container finished" podID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerID="4330764eb593ad5e8f054e44a7eeb35e7d5951ae00f0fac16403a2eb59da0b8d" exitCode=0 Oct 30 21:46:57.602172 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:57.602148 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" event={"ID":"1c87fc4f-6bb4-4cc1-a76b-fa457438619b","Type":"ContainerDied","Data":"4330764eb593ad5e8f054e44a7eeb35e7d5951ae00f0fac16403a2eb59da0b8d"} Oct 30 21:46:58.560253 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:46:58.560174 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:46:58.612169 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:58.611171 4933 generic.go:356] "Generic (PLEG): container finished" podID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerID="76cb0376a3d67dd151d3c540ed600c36ef3928e87c3e8001126c22be19c1ad80" exitCode=0 Oct 30 21:46:58.612169 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:46:58.611295 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" event={"ID":"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c","Type":"ContainerDied","Data":"76cb0376a3d67dd151d3c540ed600c36ef3928e87c3e8001126c22be19c1ad80"} Oct 30 21:47:01.641082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:01.641046 4933 generic.go:356] "Generic (PLEG): container finished" podID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerID="6831090b01afe22cdb084ae22c73a2714e065cc34d41b9d886444298bc451de2" exitCode=0 Oct 30 21:47:01.641735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:01.641704 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" event={"ID":"b58b60e2-ea09-4c93-8ede-b11251b32719","Type":"ContainerDied","Data":"6831090b01afe22cdb084ae22c73a2714e065cc34d41b9d886444298bc451de2"} Oct 30 21:47:11.752375 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:11.752316 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" event={"ID":"b58b60e2-ea09-4c93-8ede-b11251b32719","Type":"ContainerStarted","Data":"a515136134448be7c0bb290dbba2eba6d2c1c97f2d832ae957bd88653fab82f8"} Oct 30 21:47:11.756438 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:11.756408 4933 generic.go:356] "Generic (PLEG): container finished" podID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerID="ac6b75cb28f925f14b7511187180ace02e6b1a5c6cd34ec2a7a046e414eef184" exitCode=0 Oct 30 21:47:11.756596 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:11.756526 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" event={"ID":"9fb7207d-d19b-45d7-9a90-24f429eaa6b7","Type":"ContainerDied","Data":"ac6b75cb28f925f14b7511187180ace02e6b1a5c6cd34ec2a7a046e414eef184"} Oct 30 21:47:11.759544 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:11.759463 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" event={"ID":"fbc94cef-a2b1-48ad-8478-d2c880e9251a","Type":"ContainerStarted","Data":"5a780a8b8311999ad4b1352994761f278a9fc5db534192aa32ec2433d46b7845"} Oct 30 21:47:12.022228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.020083 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" podStartSLOduration=3.354644478 podStartE2EDuration="32.020067026s" totalImagesPullingTime="28.665422548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:11.898584929 +0000 UTC m=+280.579734823" watchObservedRunningTime="2026-10-30 21:47:12.020067026 +0000 UTC m=+280.701216921" Oct 30 21:47:12.142468 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.142218 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" podStartSLOduration=3.170255591 podStartE2EDuration="35.142197212s" totalImagesPullingTime="31.971941621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:12.140087367 +0000 UTC m=+280.821237262" watchObservedRunningTime="2026-10-30 21:47:12.142197212 +0000 UTC m=+280.823347106" Oct 30 21:47:12.771527 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.771332 4933 generic.go:356] "Generic (PLEG): container finished" podID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerID="5a780a8b8311999ad4b1352994761f278a9fc5db534192aa32ec2433d46b7845" exitCode=0 Oct 30 21:47:12.771527 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.771512 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" event={"ID":"fbc94cef-a2b1-48ad-8478-d2c880e9251a","Type":"ContainerDied","Data":"5a780a8b8311999ad4b1352994761f278a9fc5db534192aa32ec2433d46b7845"} Oct 30 21:47:12.773986 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.773960 4933 generic.go:356] "Generic (PLEG): container finished" podID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerID="a515136134448be7c0bb290dbba2eba6d2c1c97f2d832ae957bd88653fab82f8" exitCode=0 Oct 30 21:47:12.774400 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.774331 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" event={"ID":"b58b60e2-ea09-4c93-8ede-b11251b32719","Type":"ContainerDied","Data":"a515136134448be7c0bb290dbba2eba6d2c1c97f2d832ae957bd88653fab82f8"} Oct 30 21:47:12.935600 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.935575 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:47:12.991777 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.991751 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-bundle\") pod \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " Oct 30 21:47:12.991911 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.991839 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-util\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-util\") pod \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " Oct 30 21:47:12.991911 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.991891 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-kube-api-access-v7pjv\" (UniqueName: \"kubernetes.io/projected/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-kube-api-access-v7pjv\") pod \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\" (UID: \"9fb7207d-d19b-45d7-9a90-24f429eaa6b7\") " Oct 30 21:47:12.992296 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.992271 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-bundle" pod "9fb7207d-d19b-45d7-9a90-24f429eaa6b7" (UID: "9fb7207d-d19b-45d7-9a90-24f429eaa6b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:12.994326 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.994304 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-kube-api-access-v7pjv" pod "9fb7207d-d19b-45d7-9a90-24f429eaa6b7" (UID: "9fb7207d-d19b-45d7-9a90-24f429eaa6b7"). InnerVolumeSpecName "kube-api-access-v7pjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:12.999311 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:12.999282 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-util" pod "9fb7207d-d19b-45d7-9a90-24f429eaa6b7" (UID: "9fb7207d-d19b-45d7-9a90-24f429eaa6b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:13.093277 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.093255 4933 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-util\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:13.093368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.093281 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7pjv\" (UniqueName: \"kubernetes.io/projected/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-kube-api-access-v7pjv\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:13.093368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.093291 4933 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9fb7207d-d19b-45d7-9a90-24f429eaa6b7-bundle\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:13.780200 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.780159 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" event={"ID":"9fb7207d-d19b-45d7-9a90-24f429eaa6b7","Type":"ContainerDied","Data":"e507a127df77772851cee7da70a4b2729eabdc7fceb3bb44aca7a0d99fb22916"} Oct 30 21:47:13.780615 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.780221 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e507a127df77772851cee7da70a4b2729eabdc7fceb3bb44aca7a0d99fb22916" Oct 30 21:47:13.780615 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.780176 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71jzr8f" Oct 30 21:47:13.783898 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.783876 4933 generic.go:356] "Generic (PLEG): container finished" podID="22cc790b-d980-4c54-9f80-4c2238c05054" containerID="1696553cf9d65416a0f06d854c1a2f114ffe53f46df5d110d56210060b3a41df" exitCode=0 Oct 30 21:47:13.784026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.783961 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" event={"ID":"22cc790b-d980-4c54-9f80-4c2238c05054","Type":"ContainerDied","Data":"1696553cf9d65416a0f06d854c1a2f114ffe53f46df5d110d56210060b3a41df"} Oct 30 21:47:13.786041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.786021 4933 generic.go:356] "Generic (PLEG): container finished" podID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerID="d1865c78c1cb9acd1b277d0df41e96482bd4a86aae3d8b303b768b8880ebfc71" exitCode=0 Oct 30 21:47:13.786139 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.786084 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" event={"ID":"0d2e75f0-1f35-4323-abf7-d22f0c436455","Type":"ContainerDied","Data":"d1865c78c1cb9acd1b277d0df41e96482bd4a86aae3d8b303b768b8880ebfc71"} Oct 30 21:47:13.788109 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.788090 4933 generic.go:356] "Generic (PLEG): container finished" podID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerID="134868f85160e94d0bba36bb561922f423f0ba60ec01c16c65a29c2e05b8c5a9" exitCode=0 Oct 30 21:47:13.788212 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.788154 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" event={"ID":"1c87fc4f-6bb4-4cc1-a76b-fa457438619b","Type":"ContainerDied","Data":"134868f85160e94d0bba36bb561922f423f0ba60ec01c16c65a29c2e05b8c5a9"} Oct 30 21:47:13.790210 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.790189 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" event={"ID":"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c","Type":"ContainerStarted","Data":"4a3516c8d67979f9aace1ff9d05bff5a58a17b195b3812d87ce2b56e3948615a"} Oct 30 21:47:13.929520 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.929473 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:47:13.965448 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:13.965417 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:47:14.001534 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.001513 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-util\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-util\") pod \"b58b60e2-ea09-4c93-8ede-b11251b32719\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " Oct 30 21:47:14.001657 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.001640 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc94cef-a2b1-48ad-8478-d2c880e9251a-kube-api-access-g8tmq\" (UniqueName: \"kubernetes.io/projected/fbc94cef-a2b1-48ad-8478-d2c880e9251a-kube-api-access-g8tmq\") pod \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " Oct 30 21:47:14.001733 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.001675 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-bundle\") pod \"b58b60e2-ea09-4c93-8ede-b11251b32719\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " Oct 30 21:47:14.001733 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.001711 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-util\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-util\") pod \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " Oct 30 21:47:14.001847 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.001746 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b58b60e2-ea09-4c93-8ede-b11251b32719-kube-api-access-qrk6f\" (UniqueName: \"kubernetes.io/projected/b58b60e2-ea09-4c93-8ede-b11251b32719-kube-api-access-qrk6f\") pod \"b58b60e2-ea09-4c93-8ede-b11251b32719\" (UID: \"b58b60e2-ea09-4c93-8ede-b11251b32719\") " Oct 30 21:47:14.001847 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.001779 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-bundle\") pod \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\" (UID: \"fbc94cef-a2b1-48ad-8478-d2c880e9251a\") " Oct 30 21:47:14.002068 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.002036 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-bundle" pod "b58b60e2-ea09-4c93-8ede-b11251b32719" (UID: "b58b60e2-ea09-4c93-8ede-b11251b32719"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:14.002950 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.002921 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-bundle" pod "fbc94cef-a2b1-48ad-8478-d2c880e9251a" (UID: "fbc94cef-a2b1-48ad-8478-d2c880e9251a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:14.004579 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.004553 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc94cef-a2b1-48ad-8478-d2c880e9251a-kube-api-access-g8tmq" pod "fbc94cef-a2b1-48ad-8478-d2c880e9251a" (UID: "fbc94cef-a2b1-48ad-8478-d2c880e9251a"). InnerVolumeSpecName "kube-api-access-g8tmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:14.005516 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.005324 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b58b60e2-ea09-4c93-8ede-b11251b32719-kube-api-access-qrk6f" pod "b58b60e2-ea09-4c93-8ede-b11251b32719" (UID: "b58b60e2-ea09-4c93-8ede-b11251b32719"). InnerVolumeSpecName "kube-api-access-qrk6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:14.007765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.007746 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-util" pod "b58b60e2-ea09-4c93-8ede-b11251b32719" (UID: "b58b60e2-ea09-4c93-8ede-b11251b32719"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:14.008695 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.008668 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-util" pod "fbc94cef-a2b1-48ad-8478-d2c880e9251a" (UID: "fbc94cef-a2b1-48ad-8478-d2c880e9251a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:14.082921 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.082876 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" podStartSLOduration=3.7241258459999997 podStartE2EDuration="34.082863332s" totalImagesPullingTime="30.358737486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:14.081562453 +0000 UTC m=+282.762712345" watchObservedRunningTime="2026-10-30 21:47:14.082863332 +0000 UTC m=+282.764013225" Oct 30 21:47:14.103102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.103078 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tmq\" (UniqueName: \"kubernetes.io/projected/fbc94cef-a2b1-48ad-8478-d2c880e9251a-kube-api-access-g8tmq\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:14.103102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.103098 4933 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-bundle\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:14.103229 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.103108 4933 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-util\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:14.103229 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.103119 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrk6f\" (UniqueName: \"kubernetes.io/projected/b58b60e2-ea09-4c93-8ede-b11251b32719-kube-api-access-qrk6f\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:14.103229 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.103127 4933 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc94cef-a2b1-48ad-8478-d2c880e9251a-bundle\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:14.103229 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.103135 4933 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b58b60e2-ea09-4c93-8ede-b11251b32719-util\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:14.796028 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.796001 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" Oct 30 21:47:14.796028 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.796012 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqf7sq" event={"ID":"fbc94cef-a2b1-48ad-8478-d2c880e9251a","Type":"ContainerDied","Data":"0cb688922a80972dc95b7c81c8a388207f2c527e77ac65986aaab223e16eeb22"} Oct 30 21:47:14.796570 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.796056 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb688922a80972dc95b7c81c8a388207f2c527e77ac65986aaab223e16eeb22" Oct 30 21:47:14.798368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.798341 4933 generic.go:356] "Generic (PLEG): container finished" podID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerID="4a3516c8d67979f9aace1ff9d05bff5a58a17b195b3812d87ce2b56e3948615a" exitCode=0 Oct 30 21:47:14.798464 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.798428 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" event={"ID":"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c","Type":"ContainerDied","Data":"4a3516c8d67979f9aace1ff9d05bff5a58a17b195b3812d87ce2b56e3948615a"} Oct 30 21:47:14.800397 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.800365 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" event={"ID":"b58b60e2-ea09-4c93-8ede-b11251b32719","Type":"ContainerDied","Data":"61252a5e089c1743e09a1e3cbd3a8527b40cb0da358eff64695925de2b3d636d"} Oct 30 21:47:14.800397 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.800399 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644d5vxk" Oct 30 21:47:14.800397 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.800421 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61252a5e089c1743e09a1e3cbd3a8527b40cb0da358eff64695925de2b3d636d" Oct 30 21:47:14.945399 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.945373 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:47:14.984615 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.984597 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:47:14.987534 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:14.987518 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:47:15.010474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010454 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-bundle\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-bundle\") pod \"22cc790b-d980-4c54-9f80-4c2238c05054\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " Oct 30 21:47:15.010593 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010514 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-bundle\") pod \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " Oct 30 21:47:15.010593 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010560 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2e75f0-1f35-4323-abf7-d22f0c436455-kube-api-access-4tgnh\" (UniqueName: \"kubernetes.io/projected/0d2e75f0-1f35-4323-abf7-d22f0c436455-kube-api-access-4tgnh\") pod \"0d2e75f0-1f35-4323-abf7-d22f0c436455\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " Oct 30 21:47:15.010705 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010592 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-bundle\") pod \"0d2e75f0-1f35-4323-abf7-d22f0c436455\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " Oct 30 21:47:15.010705 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010629 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-util\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-util\") pod \"22cc790b-d980-4c54-9f80-4c2238c05054\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " Oct 30 21:47:15.010804 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010704 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cc790b-d980-4c54-9f80-4c2238c05054-kube-api-access-srsnw\" (UniqueName: \"kubernetes.io/projected/22cc790b-d980-4c54-9f80-4c2238c05054-kube-api-access-srsnw\") pod \"22cc790b-d980-4c54-9f80-4c2238c05054\" (UID: \"22cc790b-d980-4c54-9f80-4c2238c05054\") " Oct 30 21:47:15.010804 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010747 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-kube-api-access-vhf2k\" (UniqueName: \"kubernetes.io/projected/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-kube-api-access-vhf2k\") pod \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " Oct 30 21:47:15.010804 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010775 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-util\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-util\") pod \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\" (UID: \"1c87fc4f-6bb4-4cc1-a76b-fa457438619b\") " Oct 30 21:47:15.011316 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.010837 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-util\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-util\") pod \"0d2e75f0-1f35-4323-abf7-d22f0c436455\" (UID: \"0d2e75f0-1f35-4323-abf7-d22f0c436455\") " Oct 30 21:47:15.011316 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.011155 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-bundle" pod "22cc790b-d980-4c54-9f80-4c2238c05054" (UID: "22cc790b-d980-4c54-9f80-4c2238c05054"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:15.011316 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.011167 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-bundle" pod "1c87fc4f-6bb4-4cc1-a76b-fa457438619b" (UID: "1c87fc4f-6bb4-4cc1-a76b-fa457438619b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:15.011768 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.011732 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-bundle" pod "0d2e75f0-1f35-4323-abf7-d22f0c436455" (UID: "0d2e75f0-1f35-4323-abf7-d22f0c436455"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:15.013229 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.013194 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2e75f0-1f35-4323-abf7-d22f0c436455-kube-api-access-4tgnh" pod "0d2e75f0-1f35-4323-abf7-d22f0c436455" (UID: "0d2e75f0-1f35-4323-abf7-d22f0c436455"). InnerVolumeSpecName "kube-api-access-4tgnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:15.014013 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.013989 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cc790b-d980-4c54-9f80-4c2238c05054-kube-api-access-srsnw" pod "22cc790b-d980-4c54-9f80-4c2238c05054" (UID: "22cc790b-d980-4c54-9f80-4c2238c05054"). InnerVolumeSpecName "kube-api-access-srsnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:15.014314 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.014290 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-kube-api-access-vhf2k" pod "1c87fc4f-6bb4-4cc1-a76b-fa457438619b" (UID: "1c87fc4f-6bb4-4cc1-a76b-fa457438619b"). InnerVolumeSpecName "kube-api-access-vhf2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:15.017681 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.017547 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-util" pod "1c87fc4f-6bb4-4cc1-a76b-fa457438619b" (UID: "1c87fc4f-6bb4-4cc1-a76b-fa457438619b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:15.019126 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.019103 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-util" pod "22cc790b-d980-4c54-9f80-4c2238c05054" (UID: "22cc790b-d980-4c54-9f80-4c2238c05054"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:15.020560 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.020542 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-util" pod "0d2e75f0-1f35-4323-abf7-d22f0c436455" (UID: "0d2e75f0-1f35-4323-abf7-d22f0c436455"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:15.111691 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111645 4933 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-bundle\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111691 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111665 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tgnh\" (UniqueName: \"kubernetes.io/projected/0d2e75f0-1f35-4323-abf7-d22f0c436455-kube-api-access-4tgnh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111691 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111675 4933 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-bundle\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111691 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111684 4933 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-util\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111691 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111691 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srsnw\" (UniqueName: \"kubernetes.io/projected/22cc790b-d980-4c54-9f80-4c2238c05054-kube-api-access-srsnw\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111889 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111700 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhf2k\" (UniqueName: \"kubernetes.io/projected/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-kube-api-access-vhf2k\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111889 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111707 4933 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c87fc4f-6bb4-4cc1-a76b-fa457438619b-util\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111889 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111715 4933 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2e75f0-1f35-4323-abf7-d22f0c436455-util\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.111889 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.111723 4933 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22cc790b-d980-4c54-9f80-4c2238c05054-bundle\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:15.806550 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.806525 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" Oct 30 21:47:15.806970 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.806524 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gb7hc" event={"ID":"22cc790b-d980-4c54-9f80-4c2238c05054","Type":"ContainerDied","Data":"cda5f3da2208edbab0c7c21b82c3f74e6a24b6280114ced57f716775ec4e4be0"} Oct 30 21:47:15.806970 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.806650 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda5f3da2208edbab0c7c21b82c3f74e6a24b6280114ced57f716775ec4e4be0" Oct 30 21:47:15.808553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.808533 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" Oct 30 21:47:15.808669 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.808550 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cp2q6z" event={"ID":"0d2e75f0-1f35-4323-abf7-d22f0c436455","Type":"ContainerDied","Data":"821a4cba0182748e3c0a8faa5b91d1bd8e8de8b6aa5dec5090ac2afef054cffd"} Oct 30 21:47:15.808669 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.808617 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="821a4cba0182748e3c0a8faa5b91d1bd8e8de8b6aa5dec5090ac2afef054cffd" Oct 30 21:47:15.810769 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.810747 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" Oct 30 21:47:15.810868 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.810756 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mzvks" event={"ID":"1c87fc4f-6bb4-4cc1-a76b-fa457438619b","Type":"ContainerDied","Data":"28c4fc4edf8cdaa68a88545b6766f258182e9dba4442afa61d85ca882f278a9a"} Oct 30 21:47:15.810868 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.810801 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28c4fc4edf8cdaa68a88545b6766f258182e9dba4442afa61d85ca882f278a9a" Oct 30 21:47:15.925308 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:15.925286 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:47:16.019271 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.019244 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-bundle\") pod \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " Oct 30 21:47:16.019375 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.019277 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-util\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-util\") pod \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " Oct 30 21:47:16.019375 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.019354 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-kube-api-access-26s6r\" (UniqueName: \"kubernetes.io/projected/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-kube-api-access-26s6r\") pod \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\" (UID: \"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c\") " Oct 30 21:47:16.020020 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.019987 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-bundle" pod "9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" (UID: "9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:16.021457 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.021429 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-kube-api-access-26s6r" pod "9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" (UID: "9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c"). InnerVolumeSpecName "kube-api-access-26s6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:16.024760 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.024741 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-util" pod "9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" (UID: "9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:16.120502 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.120426 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26s6r\" (UniqueName: \"kubernetes.io/projected/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-kube-api-access-26s6r\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:16.120502 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.120459 4933 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-bundle\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:16.120502 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.120474 4933 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c-util\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:47:16.817107 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.817084 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" Oct 30 21:47:16.817466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.817109 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs5z25" event={"ID":"9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c","Type":"ContainerDied","Data":"c3ab26d4017e1736e791c039622c7e1b968d6ddf70ccc01de442083563bef85c"} Oct 30 21:47:16.817466 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:16.817153 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ab26d4017e1736e791c039622c7e1b968d6ddf70ccc01de442083563bef85c" Oct 30 21:47:25.292853 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.292822 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d"] Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293132 4933 cpu_manager.go:414] "containerMap: removing container" podUID="22cc790b-d980-4c54-9f80-4c2238c05054" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293143 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cc790b-d980-4c54-9f80-4c2238c05054" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293151 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293155 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293165 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293173 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293181 4933 cpu_manager.go:414] "containerMap: removing container" podUID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerName="extract" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293186 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerName="extract" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293194 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerName="extract" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293200 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerName="extract" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293208 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293212 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293222 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerName="extract" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293226 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerName="extract" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293232 4933 cpu_manager.go:414] "containerMap: removing container" podUID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293236 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293246 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293253 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293263 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293267 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293273 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293277 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293286 4933 cpu_manager.go:414] "containerMap: removing container" podUID="22cc790b-d980-4c54-9f80-4c2238c05054" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293291 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cc790b-d980-4c54-9f80-4c2238c05054" containerName="util" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293300 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293304 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293310 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerName="pull" Oct 30 21:47:25.293299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293314 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerName="pull" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293332 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerName="util" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293338 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerName="util" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293344 4933 cpu_manager.go:414] "containerMap: removing container" podUID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerName="pull" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293348 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerName="pull" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293354 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293358 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293364 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293367 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293373 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293376 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293383 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerName="util" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293387 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerName="util" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293393 4933 cpu_manager.go:414] "containerMap: removing container" podUID="22cc790b-d980-4c54-9f80-4c2238c05054" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293399 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cc790b-d980-4c54-9f80-4c2238c05054" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293451 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b58b60e2-ea09-4c93-8ede-b11251b32719" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293457 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c87fc4f-6bb4-4cc1-a76b-fa457438619b" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293462 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9901e1bd-c6a6-4c45-8480-8cb2fe1a0e4c" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293468 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cc790b-d980-4c54-9f80-4c2238c05054" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293474 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc94cef-a2b1-48ad-8478-d2c880e9251a" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293480 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2e75f0-1f35-4323-abf7-d22f0c436455" containerName="extract" Oct 30 21:47:25.294140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.293505 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb7207d-d19b-45d7-9a90-24f429eaa6b7" containerName="extract" Oct 30 21:47:25.295570 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.295555 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:25.350422 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.350394 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d"] Oct 30 21:47:25.384646 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.384620 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzss5\" (UniqueName: \"kubernetes.io/projected/30d1de33-d957-4a76-a359-9a4f183fe369-kube-api-access-kzss5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-stj5d\" (UID: \"30d1de33-d957-4a76-a359-9a4f183fe369\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:25.384782 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.384679 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30d1de33-d957-4a76-a359-9a4f183fe369-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-stj5d\" (UID: \"30d1de33-d957-4a76-a359-9a4f183fe369\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:25.398897 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.398879 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:47:25.399106 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.399090 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:47:25.399149 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.399119 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9b77c\"" Oct 30 21:47:25.485733 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.485711 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzss5\" (UniqueName: \"kubernetes.io/projected/30d1de33-d957-4a76-a359-9a4f183fe369-kube-api-access-kzss5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-stj5d\" (UID: \"30d1de33-d957-4a76-a359-9a4f183fe369\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:25.485872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.485754 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30d1de33-d957-4a76-a359-9a4f183fe369-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-stj5d\" (UID: \"30d1de33-d957-4a76-a359-9a4f183fe369\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:25.486075 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.486059 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30d1de33-d957-4a76-a359-9a4f183fe369-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-stj5d\" (UID: \"30d1de33-d957-4a76-a359-9a4f183fe369\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:25.544559 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.544504 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzss5\" (UniqueName: \"kubernetes.io/projected/30d1de33-d957-4a76-a359-9a4f183fe369-kube-api-access-kzss5\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-stj5d\" (UID: \"30d1de33-d957-4a76-a359-9a4f183fe369\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:25.605661 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:25.605640 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" Oct 30 21:47:26.039308 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:26.039284 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d"] Oct 30 21:47:26.041831 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:47:26.041807 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d1de33_d957_4a76_a359_9a4f183fe369.slice/crio-b09de25e3dff9a4118bbbb33d2b5cf8a8ced00b7421194e66e774d2121e7ea7f WatchSource:0}: Error finding container b09de25e3dff9a4118bbbb33d2b5cf8a8ced00b7421194e66e774d2121e7ea7f: Status 404 returned error can't find the container with id b09de25e3dff9a4118bbbb33d2b5cf8a8ced00b7421194e66e774d2121e7ea7f Oct 30 21:47:26.869616 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:26.869576 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" event={"ID":"30d1de33-d957-4a76-a359-9a4f183fe369","Type":"ContainerStarted","Data":"b09de25e3dff9a4118bbbb33d2b5cf8a8ced00b7421194e66e774d2121e7ea7f"} Oct 30 21:47:28.543553 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:47:28.543520 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:47:28.881503 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:28.881462 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" event={"ID":"30d1de33-d957-4a76-a359-9a4f183fe369","Type":"ContainerStarted","Data":"0e52fb8db7443513d7c4fb7dec3bcf311d6dd0e78f4d81e25645a9f2a10d60a6"} Oct 30 21:47:29.104228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:29.104180 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-stj5d" podStartSLOduration=1.76444683 podStartE2EDuration="4.104167151s" totalImagesPullingTime="2.339720321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:29.053207535 +0000 UTC m=+297.734357429" watchObservedRunningTime="2026-10-30 21:47:29.104167151 +0000 UTC m=+297.785317043" Oct 30 21:47:31.488551 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.488519 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:47:31.489559 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.489542 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:47:31.606234 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.606207 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7"] Oct 30 21:47:31.608426 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.608411 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" Oct 30 21:47:31.676795 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.676773 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rpqws\"" Oct 30 21:47:31.727397 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.727370 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:47:31.727526 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.727455 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:47:31.728986 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.728967 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7"] Oct 30 21:47:31.731735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.731717 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mt94\" (UniqueName: \"kubernetes.io/projected/2a86402f-2067-40c8-a53f-fe523aaebebc-kube-api-access-6mt94\") pod \"limitador-operator-controller-manager-5778bb4449-zgqr7\" (UID: \"2a86402f-2067-40c8-a53f-fe523aaebebc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" Oct 30 21:47:31.832451 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.832423 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mt94\" (UniqueName: \"kubernetes.io/projected/2a86402f-2067-40c8-a53f-fe523aaebebc-kube-api-access-6mt94\") pod \"limitador-operator-controller-manager-5778bb4449-zgqr7\" (UID: \"2a86402f-2067-40c8-a53f-fe523aaebebc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" Oct 30 21:47:31.896731 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.896711 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mt94\" (UniqueName: \"kubernetes.io/projected/2a86402f-2067-40c8-a53f-fe523aaebebc-kube-api-access-6mt94\") pod \"limitador-operator-controller-manager-5778bb4449-zgqr7\" (UID: \"2a86402f-2067-40c8-a53f-fe523aaebebc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" Oct 30 21:47:31.918871 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:31.918854 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" Oct 30 21:47:32.294887 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:32.294866 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7"] Oct 30 21:47:32.297034 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:47:32.297001 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a86402f_2067_40c8_a53f_fe523aaebebc.slice/crio-a447bd2815262a4271c7b6a1c46a75c89713a234924d8a61192a6e534ab97274 WatchSource:0}: Error finding container a447bd2815262a4271c7b6a1c46a75c89713a234924d8a61192a6e534ab97274: Status 404 returned error can't find the container with id a447bd2815262a4271c7b6a1c46a75c89713a234924d8a61192a6e534ab97274 Oct 30 21:47:32.899228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:32.899194 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" event={"ID":"2a86402f-2067-40c8-a53f-fe523aaebebc","Type":"ContainerStarted","Data":"a447bd2815262a4271c7b6a1c46a75c89713a234924d8a61192a6e534ab97274"} Oct 30 21:47:35.913692 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:35.913662 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" event={"ID":"2a86402f-2067-40c8-a53f-fe523aaebebc","Type":"ContainerStarted","Data":"a5736dcd01c0a1b0d2cfd78741bbaffa3f68c3ab9b1e9da16eb417e1fe68b76c"} Oct 30 21:47:35.914164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:35.913863 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" Oct 30 21:47:36.071103 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.071023 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" podStartSLOduration=1.58071575 podStartE2EDuration="5.07100968s" totalImagesPullingTime="3.49029393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:36.069025608 +0000 UTC m=+304.750175505" watchObservedRunningTime="2026-10-30 21:47:36.07100968 +0000 UTC m=+304.752159570" Oct 30 21:47:36.437360 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.437291 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zj45z"] Oct 30 21:47:36.439519 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.439483 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" Oct 30 21:47:36.499098 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.499072 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mffjg\"" Oct 30 21:47:36.551088 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.551058 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zj45z"] Oct 30 21:47:36.573694 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.573669 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxb4\" (UniqueName: \"kubernetes.io/projected/00f89aea-4186-454a-a88b-91ac4a55b503-kube-api-access-lsxb4\") pod \"authorino-operator-7b7bb4bc77-zj45z\" (UID: \"00f89aea-4186-454a-a88b-91ac4a55b503\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" Oct 30 21:47:36.674480 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.674458 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxb4\" (UniqueName: \"kubernetes.io/projected/00f89aea-4186-454a-a88b-91ac4a55b503-kube-api-access-lsxb4\") pod \"authorino-operator-7b7bb4bc77-zj45z\" (UID: \"00f89aea-4186-454a-a88b-91ac4a55b503\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" Oct 30 21:47:36.737139 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.737084 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxb4\" (UniqueName: \"kubernetes.io/projected/00f89aea-4186-454a-a88b-91ac4a55b503-kube-api-access-lsxb4\") pod \"authorino-operator-7b7bb4bc77-zj45z\" (UID: \"00f89aea-4186-454a-a88b-91ac4a55b503\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" Oct 30 21:47:36.751027 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:36.751009 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" Oct 30 21:47:37.133998 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:37.133974 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zj45z"] Oct 30 21:47:37.136289 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:47:37.136263 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f89aea_4186_454a_a88b_91ac4a55b503.slice/crio-7dd48b78a0616552821093638ec3485d0b1890b254164025a5759b525c9b1d8a WatchSource:0}: Error finding container 7dd48b78a0616552821093638ec3485d0b1890b254164025a5759b525c9b1d8a: Status 404 returned error can't find the container with id 7dd48b78a0616552821093638ec3485d0b1890b254164025a5759b525c9b1d8a Oct 30 21:47:37.138568 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:37.138553 4933 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:47:37.924446 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:37.924387 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" event={"ID":"00f89aea-4186-454a-a88b-91ac4a55b503","Type":"ContainerStarted","Data":"7dd48b78a0616552821093638ec3485d0b1890b254164025a5759b525c9b1d8a"} Oct 30 21:47:39.371478 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.371455 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg"] Oct 30 21:47:39.374410 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.374393 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.435759 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.435736 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:47:39.435898 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.435761 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:47:39.435898 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.435831 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:47:39.436024 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.435897 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:47:39.436024 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.435935 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:47:39.437451 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.437438 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-w7mjw\"" Oct 30 21:47:39.489415 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.489386 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg"] Oct 30 21:47:39.497439 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.497413 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.497575 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.497455 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-tmp\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.497575 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.497482 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcj9j\" (UniqueName: \"kubernetes.io/projected/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-kube-api-access-vcj9j\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.497575 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.497559 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-manifests\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.497695 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.497654 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-metrics-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.497774 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.497757 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.599191 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.599122 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.599191 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.599173 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.599361 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.599201 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-tmp\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.599402 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.599374 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcj9j\" (UniqueName: \"kubernetes.io/projected/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-kube-api-access-vcj9j\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.599445 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.599422 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-manifests\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.599536 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.599455 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-metrics-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.599809 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.599779 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-manifests\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.601760 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.601736 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-tmp\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.601896 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.601874 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-webhook-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.601961 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.601880 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.601999 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.601988 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-metrics-cert\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.658282 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.658258 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcj9j\" (UniqueName: \"kubernetes.io/projected/d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e-kube-api-access-vcj9j\") pod \"opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg\" (UID: \"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.687544 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.687521 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:39.936690 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.936611 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" event={"ID":"00f89aea-4186-454a-a88b-91ac4a55b503","Type":"ContainerStarted","Data":"5b00abd4d55bc9d6f0e92ceacc4beab3da14de583a6c91c727625a693dac708d"} Oct 30 21:47:39.936690 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:39.936686 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" Oct 30 21:47:40.064559 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:40.064510 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" podStartSLOduration=1.876047888 podStartE2EDuration="4.0644796s" totalImagesPullingTime="2.188431712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:40.063837081 +0000 UTC m=+308.744986974" watchObservedRunningTime="2026-10-30 21:47:40.0644796 +0000 UTC m=+308.745629493" Oct 30 21:47:40.223334 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:40.223128 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg"] Oct 30 21:47:40.226103 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:47:40.226073 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81bf7a0_bf59_48a6_8ef6_87cc881c9d0e.slice/crio-9c0fded48d7b0536a98713a97dcf64aa29fa0c2aae97e2053c743796c0f7eeec WatchSource:0}: Error finding container 9c0fded48d7b0536a98713a97dcf64aa29fa0c2aae97e2053c743796c0f7eeec: Status 404 returned error can't find the container with id 9c0fded48d7b0536a98713a97dcf64aa29fa0c2aae97e2053c743796c0f7eeec Oct 30 21:47:40.942573 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:40.942535 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" event={"ID":"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e","Type":"ContainerStarted","Data":"9c0fded48d7b0536a98713a97dcf64aa29fa0c2aae97e2053c743796c0f7eeec"} Oct 30 21:47:41.538275 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.536647 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm"] Oct 30 21:47:41.563661 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.563630 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" Oct 30 21:47:41.598844 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.598778 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm"] Oct 30 21:47:41.615214 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.615189 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjk4w\" (UniqueName: \"kubernetes.io/projected/c1b48785-35d1-41cf-b032-b74d4d74bfee-kube-api-access-xjk4w\") pod \"dns-operator-controller-manager-8557cbdb84-tbpwm\" (UID: \"c1b48785-35d1-41cf-b032-b74d4d74bfee\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" Oct 30 21:47:41.619518 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.619475 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:47:41.660449 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.660420 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7z8jp\"" Oct 30 21:47:41.716974 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.716946 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjk4w\" (UniqueName: \"kubernetes.io/projected/c1b48785-35d1-41cf-b032-b74d4d74bfee-kube-api-access-xjk4w\") pod \"dns-operator-controller-manager-8557cbdb84-tbpwm\" (UID: \"c1b48785-35d1-41cf-b032-b74d4d74bfee\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" Oct 30 21:47:41.782757 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.782716 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjk4w\" (UniqueName: \"kubernetes.io/projected/c1b48785-35d1-41cf-b032-b74d4d74bfee-kube-api-access-xjk4w\") pod \"dns-operator-controller-manager-8557cbdb84-tbpwm\" (UID: \"c1b48785-35d1-41cf-b032-b74d4d74bfee\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" Oct 30 21:47:41.896309 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:41.896231 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" Oct 30 21:47:42.306228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:42.306202 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm"] Oct 30 21:47:42.308081 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:47:42.308046 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b48785_35d1_41cf_b032_b74d4d74bfee.slice/crio-073d2af342e166e57957fb64836ad33795eb5e013618339f9ccc51740692ee40 WatchSource:0}: Error finding container 073d2af342e166e57957fb64836ad33795eb5e013618339f9ccc51740692ee40: Status 404 returned error can't find the container with id 073d2af342e166e57957fb64836ad33795eb5e013618339f9ccc51740692ee40 Oct 30 21:47:42.956575 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:42.956542 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" event={"ID":"c1b48785-35d1-41cf-b032-b74d4d74bfee","Type":"ContainerStarted","Data":"073d2af342e166e57957fb64836ad33795eb5e013618339f9ccc51740692ee40"} Oct 30 21:47:44.972919 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:44.972827 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" event={"ID":"c1b48785-35d1-41cf-b032-b74d4d74bfee","Type":"ContainerStarted","Data":"ca36d9d760629963156c3d6ecd01eda963518229da68d652e78900b5931c6720"} Oct 30 21:47:44.973279 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:44.972949 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" Oct 30 21:47:44.974718 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:44.974689 4933 generic.go:356] "Generic (PLEG): container finished" podID="d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e" containerID="08beec0694eaea5a7ccd6442e37d27bb9fbd5cee065ce6c773b2c63efd19e5b1" exitCode=0 Oct 30 21:47:44.974839 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:44.974746 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" event={"ID":"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e","Type":"ContainerDied","Data":"08beec0694eaea5a7ccd6442e37d27bb9fbd5cee065ce6c773b2c63efd19e5b1"} Oct 30 21:47:45.096449 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:45.096401 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" podStartSLOduration=1.8762032899999999 podStartE2EDuration="4.096389618s" totalImagesPullingTime="2.220186328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:45.095446944 +0000 UTC m=+313.776596840" watchObservedRunningTime="2026-10-30 21:47:45.096389618 +0000 UTC m=+313.777539510" Oct 30 21:47:45.981855 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:45.981824 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" event={"ID":"d81bf7a0-bf59-48a6-8ef6-87cc881c9d0e","Type":"ContainerStarted","Data":"8af53c0b770c2566c7e2f3a801eb39442a0e552e37903d87d7b57b9d642de773"} Oct 30 21:47:45.982225 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:45.981909 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:46.216740 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:46.216647 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" podStartSLOduration=2.137726395 podStartE2EDuration="7.216633229s" totalImagesPullingTime="4.534516939s" totalInitContainerRuntime="544.389895ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:46.1652824 +0000 UTC m=+314.846432306" watchObservedRunningTime="2026-10-30 21:47:46.216633229 +0000 UTC m=+314.897783122" Oct 30 21:47:46.920290 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:46.920255 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-zgqr7" Oct 30 21:47:48.285128 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.285089 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz"] Oct 30 21:47:48.287648 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.287627 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:48.329895 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.329865 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz"] Oct 30 21:47:48.378785 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.378758 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vp8s4\"" Oct 30 21:47:48.379378 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.379361 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/7db7387c-8320-4580-a561-121bd21ff2d9-kube-api-access-92c8f\") pod \"kuadrant-operator-controller-manager-678d87766f-w57qz\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:48.379438 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.379393 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db7387c-8320-4580-a561-121bd21ff2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w57qz\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:48.480198 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.480172 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/7db7387c-8320-4580-a561-121bd21ff2d9-kube-api-access-92c8f\") pod \"kuadrant-operator-controller-manager-678d87766f-w57qz\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:48.480304 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.480209 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db7387c-8320-4580-a561-121bd21ff2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w57qz\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:48.480558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.480543 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db7387c-8320-4580-a561-121bd21ff2d9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w57qz\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:48.545690 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.545636 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/7db7387c-8320-4580-a561-121bd21ff2d9-kube-api-access-92c8f\") pod \"kuadrant-operator-controller-manager-678d87766f-w57qz\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:48.599026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:48.599002 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:49.068962 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:49.068935 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz"] Oct 30 21:47:49.070746 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:47:49.070720 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db7387c_8320_4580_a561_121bd21ff2d9.slice/crio-0123b693f037f665c6cb053d2060e40dcf13fbec36459cd14a6d9ea5e59dcc36 WatchSource:0}: Error finding container 0123b693f037f665c6cb053d2060e40dcf13fbec36459cd14a6d9ea5e59dcc36: Status 404 returned error can't find the container with id 0123b693f037f665c6cb053d2060e40dcf13fbec36459cd14a6d9ea5e59dcc36 Oct 30 21:47:50.002444 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:50.002415 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" event={"ID":"7db7387c-8320-4580-a561-121bd21ff2d9","Type":"ContainerStarted","Data":"0123b693f037f665c6cb053d2060e40dcf13fbec36459cd14a6d9ea5e59dcc36"} Oct 30 21:47:50.945589 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:50.945562 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zj45z" Oct 30 21:47:55.985191 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:55.985153 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tbpwm" Oct 30 21:47:56.990163 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:56.990135 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c8dcc6d7b-w2dfg" Oct 30 21:47:58.547234 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:47:58.547175 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:47:59.045973 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:59.045940 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" event={"ID":"7db7387c-8320-4580-a561-121bd21ff2d9","Type":"ContainerStarted","Data":"2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520"} Oct 30 21:47:59.046161 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:59.046080 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:47:59.267779 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:47:59.267726 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" podStartSLOduration=1.7312767980000001 podStartE2EDuration="11.267709563s" totalImagesPullingTime="9.536432765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:59.266381509 +0000 UTC m=+327.947531402" watchObservedRunningTime="2026-10-30 21:47:59.267709563 +0000 UTC m=+327.948859457" Oct 30 21:48:10.053254 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:10.053223 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 21:48:17.318045 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.318013 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh"] Oct 30 21:48:17.323714 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.323692 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.374281 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.374251 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh"] Oct 30 21:48:17.391326 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391294 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391326 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391325 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391477 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391433 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391570 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391552 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391612 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391596 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391656 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391641 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391733 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391719 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391773 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391758 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7628\" (UniqueName: \"kubernetes.io/projected/8ec5f880-07fb-4331-ad83-e193ebbf62f2-kube-api-access-n7628\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.391824 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.391777 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.422322 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.422301 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-v5vhq\"" Oct 30 21:48:17.492969 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.492936 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.492973 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493005 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493048 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493078 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493349 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493190 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7628\" (UniqueName: \"kubernetes.io/projected/8ec5f880-07fb-4331-ad83-e193ebbf62f2-kube-api-access-n7628\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493349 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493242 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493349 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493331 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493590 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493572 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493667 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493629 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493667 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493643 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493779 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493694 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493779 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493700 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.493899 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.493878 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.495685 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.495659 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.495782 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.495727 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.550470 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.550447 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec5f880-07fb-4331-ad83-e193ebbf62f2-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.550579 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.550563 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7628\" (UniqueName: \"kubernetes.io/projected/8ec5f880-07fb-4331-ad83-e193ebbf62f2-kube-api-access-n7628\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d97d4qh\" (UID: \"8ec5f880-07fb-4331-ad83-e193ebbf62f2\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:17.634875 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:17.634816 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:18.062910 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.062888 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh"] Oct 30 21:48:18.064521 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:48:18.064481 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec5f880_07fb_4331_ad83_e193ebbf62f2.slice/crio-3c7f5b8ec2d7b69f24a723fc72eba8804a549fffd0011fbd0fd637a25f99f33c WatchSource:0}: Error finding container 3c7f5b8ec2d7b69f24a723fc72eba8804a549fffd0011fbd0fd637a25f99f33c: Status 404 returned error can't find the container with id 3c7f5b8ec2d7b69f24a723fc72eba8804a549fffd0011fbd0fd637a25f99f33c Oct 30 21:48:18.066441 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.066410 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:18.066517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.066479 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:18.066553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.066526 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:48:18.137179 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.137139 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" event={"ID":"8ec5f880-07fb-4331-ad83-e193ebbf62f2","Type":"ContainerStarted","Data":"161bcb00ce3c0f05d6346782e06618ffceb79ff13a4ac2edaf37be553afc9b00"} Oct 30 21:48:18.137327 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.137213 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" event={"ID":"8ec5f880-07fb-4331-ad83-e193ebbf62f2","Type":"ContainerStarted","Data":"3c7f5b8ec2d7b69f24a723fc72eba8804a549fffd0011fbd0fd637a25f99f33c"} Oct 30 21:48:18.327045 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.326943 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" podStartSLOduration=1.3269252200000001 podStartE2EDuration="1.32692522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:18.320475424 +0000 UTC m=+347.001625317" watchObservedRunningTime="2026-10-30 21:48:18.32692522 +0000 UTC m=+347.008075116" Oct 30 21:48:18.635087 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:18.635014 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:19.640125 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:19.640091 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:20.146670 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:20.146639 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:20.147609 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:20.147592 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d97d4qh" Oct 30 21:48:21.096244 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.096215 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-skmdj"] Oct 30 21:48:21.098564 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.098548 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-skmdj" Oct 30 21:48:21.122362 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.122336 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lh9\" (UniqueName: \"kubernetes.io/projected/474ad007-ec0d-4447-b852-f1697b6e1296-kube-api-access-r6lh9\") pod \"authorino-57f7f8659f-skmdj\" (UID: \"474ad007-ec0d-4447-b852-f1697b6e1296\") " pod="kuadrant-system/authorino-57f7f8659f-skmdj" Oct 30 21:48:21.158285 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.158264 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-skmdj"] Oct 30 21:48:21.206411 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.206386 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rj7nv\"" Oct 30 21:48:21.223020 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.222992 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lh9\" (UniqueName: \"kubernetes.io/projected/474ad007-ec0d-4447-b852-f1697b6e1296-kube-api-access-r6lh9\") pod \"authorino-57f7f8659f-skmdj\" (UID: \"474ad007-ec0d-4447-b852-f1697b6e1296\") " pod="kuadrant-system/authorino-57f7f8659f-skmdj" Oct 30 21:48:21.280671 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.280650 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lh9\" (UniqueName: \"kubernetes.io/projected/474ad007-ec0d-4447-b852-f1697b6e1296-kube-api-access-r6lh9\") pod \"authorino-57f7f8659f-skmdj\" (UID: \"474ad007-ec0d-4447-b852-f1697b6e1296\") " pod="kuadrant-system/authorino-57f7f8659f-skmdj" Oct 30 21:48:21.419817 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.419766 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-skmdj" Oct 30 21:48:21.787782 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:21.787759 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-skmdj"] Oct 30 21:48:21.789540 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:48:21.789516 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod474ad007_ec0d_4447_b852_f1697b6e1296.slice/crio-52c8098fbc45c022e5d1bce100381c874c110eab5d24ee1a3c7d9b83eeccc7ac WatchSource:0}: Error finding container 52c8098fbc45c022e5d1bce100381c874c110eab5d24ee1a3c7d9b83eeccc7ac: Status 404 returned error can't find the container with id 52c8098fbc45c022e5d1bce100381c874c110eab5d24ee1a3c7d9b83eeccc7ac Oct 30 21:48:22.157854 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.157776 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-skmdj" event={"ID":"474ad007-ec0d-4447-b852-f1697b6e1296","Type":"ContainerStarted","Data":"52c8098fbc45c022e5d1bce100381c874c110eab5d24ee1a3c7d9b83eeccc7ac"} Oct 30 21:48:22.642700 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.642667 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-sphm5"] Oct 30 21:48:22.645590 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.645570 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:48:22.645704 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.645686 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" Oct 30 21:48:22.647527 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.647508 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.702478 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.702454 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-sphm5"] Oct 30 21:48:22.703521 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.703481 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:48:22.705593 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.705571 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:48:22.733026 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.733004 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2klq\" (UniqueName: \"kubernetes.io/projected/4d577a18-2871-4312-97f6-8b1e7ecd1fb8-kube-api-access-b2klq\") pod \"limitador-limitador-7c9b87d574-h5bzd\" (UID: \"4d577a18-2871-4312-97f6-8b1e7ecd1fb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.733138 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.733058 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d577a18-2871-4312-97f6-8b1e7ecd1fb8-config-file\") pod \"limitador-limitador-7c9b87d574-h5bzd\" (UID: \"4d577a18-2871-4312-97f6-8b1e7ecd1fb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.733188 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.733143 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fml\" (UniqueName: \"kubernetes.io/projected/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47-kube-api-access-t4fml\") pod \"authorino-56c6c9ddd5-sphm5\" (UID: \"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47\") " pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" Oct 30 21:48:22.778127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.778107 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6d9db\"" Oct 30 21:48:22.834477 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.834443 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2klq\" (UniqueName: \"kubernetes.io/projected/4d577a18-2871-4312-97f6-8b1e7ecd1fb8-kube-api-access-b2klq\") pod \"limitador-limitador-7c9b87d574-h5bzd\" (UID: \"4d577a18-2871-4312-97f6-8b1e7ecd1fb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.834591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.834535 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d577a18-2871-4312-97f6-8b1e7ecd1fb8-config-file\") pod \"limitador-limitador-7c9b87d574-h5bzd\" (UID: \"4d577a18-2871-4312-97f6-8b1e7ecd1fb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.834591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.834564 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fml\" (UniqueName: \"kubernetes.io/projected/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47-kube-api-access-t4fml\") pod \"authorino-56c6c9ddd5-sphm5\" (UID: \"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47\") " pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" Oct 30 21:48:22.835139 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.835122 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4d577a18-2871-4312-97f6-8b1e7ecd1fb8-config-file\") pod \"limitador-limitador-7c9b87d574-h5bzd\" (UID: \"4d577a18-2871-4312-97f6-8b1e7ecd1fb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.892420 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.892398 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fml\" (UniqueName: \"kubernetes.io/projected/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47-kube-api-access-t4fml\") pod \"authorino-56c6c9ddd5-sphm5\" (UID: \"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47\") " pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" Oct 30 21:48:22.892775 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.892738 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2klq\" (UniqueName: \"kubernetes.io/projected/4d577a18-2871-4312-97f6-8b1e7ecd1fb8-kube-api-access-b2klq\") pod \"limitador-limitador-7c9b87d574-h5bzd\" (UID: \"4d577a18-2871-4312-97f6-8b1e7ecd1fb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.958147 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.958126 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" Oct 30 21:48:22.963856 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.963841 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:22.968099 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:22.968067 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:48:23.331453 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:23.331431 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-sphm5"] Oct 30 21:48:23.333692 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:48:23.333669 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa6bebf6_77de_4664_84b8_ce0f0a6e8b47.slice/crio-9f54d15e08cf878d45e2e667729f1987ccd7ce88c11d8238c99017be67e02eed WatchSource:0}: Error finding container 9f54d15e08cf878d45e2e667729f1987ccd7ce88c11d8238c99017be67e02eed: Status 404 returned error can't find the container with id 9f54d15e08cf878d45e2e667729f1987ccd7ce88c11d8238c99017be67e02eed Oct 30 21:48:23.384154 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:23.381452 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:48:23.387963 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:48:23.387933 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d577a18_2871_4312_97f6_8b1e7ecd1fb8.slice/crio-d7eb34656b20646946ea25e273e3ff37e8f74f466bcf08fb648fb575aff84b0e WatchSource:0}: Error finding container d7eb34656b20646946ea25e273e3ff37e8f74f466bcf08fb648fb575aff84b0e: Status 404 returned error can't find the container with id d7eb34656b20646946ea25e273e3ff37e8f74f466bcf08fb648fb575aff84b0e Oct 30 21:48:24.170938 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:24.170900 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" event={"ID":"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47","Type":"ContainerStarted","Data":"9f54d15e08cf878d45e2e667729f1987ccd7ce88c11d8238c99017be67e02eed"} Oct 30 21:48:24.172221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:24.172192 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" event={"ID":"4d577a18-2871-4312-97f6-8b1e7ecd1fb8","Type":"ContainerStarted","Data":"d7eb34656b20646946ea25e273e3ff37e8f74f466bcf08fb648fb575aff84b0e"} Oct 30 21:48:25.178975 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:25.178935 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" event={"ID":"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47","Type":"ContainerStarted","Data":"11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7"} Oct 30 21:48:25.180966 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:25.180936 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-skmdj" event={"ID":"474ad007-ec0d-4447-b852-f1697b6e1296","Type":"ContainerStarted","Data":"84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f"} Oct 30 21:48:25.292289 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:25.292251 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" podStartSLOduration=1.872065391 podStartE2EDuration="3.292237683s" totalImagesPullingTime="1.420172292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:25.291579748 +0000 UTC m=+353.972729641" watchObservedRunningTime="2026-10-30 21:48:25.292237683 +0000 UTC m=+353.973387576" Oct 30 21:48:25.339925 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:25.339898 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-skmdj"] Oct 30 21:48:25.411205 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:25.411095 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-skmdj" podStartSLOduration=1.979845171 podStartE2EDuration="4.411069843s" totalImagesPullingTime="2.431224672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:25.408928871 +0000 UTC m=+354.090078765" watchObservedRunningTime="2026-10-30 21:48:25.411069843 +0000 UTC m=+354.092219737" Oct 30 21:48:26.187658 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:26.187623 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" event={"ID":"4d577a18-2871-4312-97f6-8b1e7ecd1fb8","Type":"ContainerStarted","Data":"53fda162d9b0bebc31800549811601ad4d2279ad145e880b5f52d5587c303415"} Oct 30 21:48:26.304039 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:26.303990 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" podStartSLOduration=2.5529906799999997 podStartE2EDuration="4.30397646s" totalImagesPullingTime="1.75098578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:26.303668874 +0000 UTC m=+354.984818767" watchObservedRunningTime="2026-10-30 21:48:26.30397646 +0000 UTC m=+354.985126352" Oct 30 21:48:27.192923 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:27.192879 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-skmdj" podUID="474ad007-ec0d-4447-b852-f1697b6e1296" containerName="authorino" containerID="cri-o://84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f" gracePeriod=30 Oct 30 21:48:27.193388 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:27.193061 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:27.433439 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:27.433419 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-skmdj" Oct 30 21:48:27.474631 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:27.474572 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474ad007-ec0d-4447-b852-f1697b6e1296-kube-api-access-r6lh9\" (UniqueName: \"kubernetes.io/projected/474ad007-ec0d-4447-b852-f1697b6e1296-kube-api-access-r6lh9\") pod \"474ad007-ec0d-4447-b852-f1697b6e1296\" (UID: \"474ad007-ec0d-4447-b852-f1697b6e1296\") " Oct 30 21:48:27.476821 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:27.476792 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474ad007-ec0d-4447-b852-f1697b6e1296-kube-api-access-r6lh9" pod "474ad007-ec0d-4447-b852-f1697b6e1296" (UID: "474ad007-ec0d-4447-b852-f1697b6e1296"). InnerVolumeSpecName "kube-api-access-r6lh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:27.576019 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:27.575987 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lh9\" (UniqueName: \"kubernetes.io/projected/474ad007-ec0d-4447-b852-f1697b6e1296-kube-api-access-r6lh9\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:48:28.198792 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.198757 4933 generic.go:356] "Generic (PLEG): container finished" podID="474ad007-ec0d-4447-b852-f1697b6e1296" containerID="84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f" exitCode=0 Oct 30 21:48:28.199266 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.198797 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-skmdj" Oct 30 21:48:28.199266 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.198853 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-skmdj" event={"ID":"474ad007-ec0d-4447-b852-f1697b6e1296","Type":"ContainerDied","Data":"84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f"} Oct 30 21:48:28.199266 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.198894 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-skmdj" event={"ID":"474ad007-ec0d-4447-b852-f1697b6e1296","Type":"ContainerDied","Data":"52c8098fbc45c022e5d1bce100381c874c110eab5d24ee1a3c7d9b83eeccc7ac"} Oct 30 21:48:28.199266 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.198910 4933 scope.go:122] "RemoveContainer" containerID="84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f" Oct 30 21:48:28.207553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.207536 4933 scope.go:122] "RemoveContainer" containerID="84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f" Oct 30 21:48:28.207808 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:48:28.207790 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f\": container with ID starting with 84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f not found: ID does not exist" containerID="84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f" Oct 30 21:48:28.207855 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.207817 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f"} err="failed to get container status \"84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f\": rpc error: code = NotFound desc = could not find container \"84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f\": container with ID starting with 84307e12cd57cf3ae34ade02bddb94841cc8da79e750ffcdab39d3b6ddc4354f not found: ID does not exist" Oct 30 21:48:28.367281 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.367256 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-skmdj"] Oct 30 21:48:28.369041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:28.369023 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-skmdj"] Oct 30 21:48:28.541669 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:48:28.541627 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:48:29.469058 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:29.469026 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474ad007-ec0d-4447-b852-f1697b6e1296" path="/var/lib/kubelet/pods/474ad007-ec0d-4447-b852-f1697b6e1296/volumes" Oct 30 21:48:38.200658 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:48:38.200628 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h5bzd" Oct 30 21:48:58.543071 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:48:58.543031 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:49:28.542829 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:49:28.542742 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:49:50.736036 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:50.735987 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:49:52.581437 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.581402 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-86z25"] Oct 30 21:49:52.581955 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.581939 4933 cpu_manager.go:414] "containerMap: removing container" podUID="474ad007-ec0d-4447-b852-f1697b6e1296" containerName="authorino" Oct 30 21:49:52.582000 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.581964 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474ad007-ec0d-4447-b852-f1697b6e1296" containerName="authorino" Oct 30 21:49:52.582034 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.582021 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="474ad007-ec0d-4447-b852-f1697b6e1296" containerName="authorino" Oct 30 21:49:52.585005 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.584992 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.621804 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.621784 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb1bfed-1923-4fb4-a7a9-7c34ae56655a-cert\") pod \"kserve-controller-manager-776cc66d47-86z25\" (UID: \"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.621910 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.621836 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6cg\" (UniqueName: \"kubernetes.io/projected/4bb1bfed-1923-4fb4-a7a9-7c34ae56655a-kube-api-access-zg6cg\") pod \"kserve-controller-manager-776cc66d47-86z25\" (UID: \"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.659600 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.655068 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-86z25"] Oct 30 21:49:52.703643 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.703612 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:49:52.703828 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.703695 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-s2r4b\"" Oct 30 21:49:52.703887 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.703828 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:49:52.703994 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.703980 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:49:52.706171 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.706147 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl"] Oct 30 21:49:52.709758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.709735 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:52.722402 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.722374 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb1bfed-1923-4fb4-a7a9-7c34ae56655a-cert\") pod \"kserve-controller-manager-776cc66d47-86z25\" (UID: \"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.722553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.722458 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6cg\" (UniqueName: \"kubernetes.io/projected/4bb1bfed-1923-4fb4-a7a9-7c34ae56655a-kube-api-access-zg6cg\") pod \"kserve-controller-manager-776cc66d47-86z25\" (UID: \"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.725230 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.725209 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bb1bfed-1923-4fb4-a7a9-7c34ae56655a-cert\") pod \"kserve-controller-manager-776cc66d47-86z25\" (UID: \"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.760512 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.759115 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl"] Oct 30 21:49:52.763839 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.763818 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:49:52.763973 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.763882 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-x2m7w\"" Oct 30 21:49:52.780750 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.780731 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6cg\" (UniqueName: \"kubernetes.io/projected/4bb1bfed-1923-4fb4-a7a9-7c34ae56655a-kube-api-access-zg6cg\") pod \"kserve-controller-manager-776cc66d47-86z25\" (UID: \"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.802349 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.802324 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jpwxz"] Oct 30 21:49:52.805862 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.805845 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:52.823866 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.823843 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5ct\" (UniqueName: \"kubernetes.io/projected/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-kube-api-access-6k5ct\") pod \"llmisvc-controller-manager-869d5dcc5d-56rkl\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:52.823964 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.823952 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-56rkl\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:52.864049 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.863978 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jpwxz"] Oct 30 21:49:52.864863 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.864824 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:49:52.865150 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.865128 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s98q5\"" Oct 30 21:49:52.897126 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.897105 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:52.925363 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.925342 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrts\" (UniqueName: \"kubernetes.io/projected/387430b1-4ca6-4b40-80a4-6e64323f2f40-kube-api-access-gvrts\") pod \"model-serving-api-75bd89fdfc-jpwxz\" (UID: \"387430b1-4ca6-4b40-80a4-6e64323f2f40\") " pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:52.925444 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.925375 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5ct\" (UniqueName: \"kubernetes.io/projected/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-kube-api-access-6k5ct\") pod \"llmisvc-controller-manager-869d5dcc5d-56rkl\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:52.925444 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.925411 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387430b1-4ca6-4b40-80a4-6e64323f2f40-tls-certs\") pod \"model-serving-api-75bd89fdfc-jpwxz\" (UID: \"387430b1-4ca6-4b40-80a4-6e64323f2f40\") " pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:52.925739 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.925723 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-56rkl\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:52.928123 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.928103 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-56rkl\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:52.983820 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:52.983799 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5ct\" (UniqueName: \"kubernetes.io/projected/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-kube-api-access-6k5ct\") pod \"llmisvc-controller-manager-869d5dcc5d-56rkl\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:53.023301 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.023278 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:53.026754 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.026727 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387430b1-4ca6-4b40-80a4-6e64323f2f40-tls-certs\") pod \"model-serving-api-75bd89fdfc-jpwxz\" (UID: \"387430b1-4ca6-4b40-80a4-6e64323f2f40\") " pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:53.026879 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.026861 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrts\" (UniqueName: \"kubernetes.io/projected/387430b1-4ca6-4b40-80a4-6e64323f2f40-kube-api-access-gvrts\") pod \"model-serving-api-75bd89fdfc-jpwxz\" (UID: \"387430b1-4ca6-4b40-80a4-6e64323f2f40\") " pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:53.029068 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.029018 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387430b1-4ca6-4b40-80a4-6e64323f2f40-tls-certs\") pod \"model-serving-api-75bd89fdfc-jpwxz\" (UID: \"387430b1-4ca6-4b40-80a4-6e64323f2f40\") " pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:53.085456 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.085401 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrts\" (UniqueName: \"kubernetes.io/projected/387430b1-4ca6-4b40-80a4-6e64323f2f40-kube-api-access-gvrts\") pod \"model-serving-api-75bd89fdfc-jpwxz\" (UID: \"387430b1-4ca6-4b40-80a4-6e64323f2f40\") " pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:53.118875 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.118819 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:53.304428 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.304400 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-86z25"] Oct 30 21:49:53.305221 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:49:53.305194 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb1bfed_1923_4fb4_a7a9_7c34ae56655a.slice/crio-fd414599626b9ae050466d1e0066b700743c2668cff55f22e4ba6c52844710dd WatchSource:0}: Error finding container fd414599626b9ae050466d1e0066b700743c2668cff55f22e4ba6c52844710dd: Status 404 returned error can't find the container with id fd414599626b9ae050466d1e0066b700743c2668cff55f22e4ba6c52844710dd Oct 30 21:49:53.421865 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.421838 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl"] Oct 30 21:49:53.422681 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:49:53.422654 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1c9c99e_1ac0_4a2c_b722_0acc313ced68.slice/crio-a0c5e095bf735bcf025a704d1b48bed2dbdc20daffcd9cb6dd41b1936730a328 WatchSource:0}: Error finding container a0c5e095bf735bcf025a704d1b48bed2dbdc20daffcd9cb6dd41b1936730a328: Status 404 returned error can't find the container with id a0c5e095bf735bcf025a704d1b48bed2dbdc20daffcd9cb6dd41b1936730a328 Oct 30 21:49:53.520867 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.520846 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jpwxz"] Oct 30 21:49:53.522380 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:49:53.522350 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387430b1_4ca6_4b40_80a4_6e64323f2f40.slice/crio-3904037854b7d56b52c0df6f2bb5e83d7752bca8373a7ac09fd7a36b2179ec4d WatchSource:0}: Error finding container 3904037854b7d56b52c0df6f2bb5e83d7752bca8373a7ac09fd7a36b2179ec4d: Status 404 returned error can't find the container with id 3904037854b7d56b52c0df6f2bb5e83d7752bca8373a7ac09fd7a36b2179ec4d Oct 30 21:49:53.612127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.612090 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" event={"ID":"b1c9c99e-1ac0-4a2c-b722-0acc313ced68","Type":"ContainerStarted","Data":"a0c5e095bf735bcf025a704d1b48bed2dbdc20daffcd9cb6dd41b1936730a328"} Oct 30 21:49:53.613364 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.613342 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" event={"ID":"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a","Type":"ContainerStarted","Data":"fd414599626b9ae050466d1e0066b700743c2668cff55f22e4ba6c52844710dd"} Oct 30 21:49:53.614606 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:53.614584 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" event={"ID":"387430b1-4ca6-4b40-80a4-6e64323f2f40","Type":"ContainerStarted","Data":"3904037854b7d56b52c0df6f2bb5e83d7752bca8373a7ac09fd7a36b2179ec4d"} Oct 30 21:49:55.056877 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.056846 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pr5pp"] Oct 30 21:49:55.060885 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.060870 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" Oct 30 21:49:55.098260 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.098206 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pr5pp"] Oct 30 21:49:55.175080 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.174892 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrd5\" (UniqueName: \"kubernetes.io/projected/159683c4-0929-4a72-890d-668ebd15c0da-kube-api-access-sdrd5\") pod \"authorino-6f6944bcdd-pr5pp\" (UID: \"159683c4-0929-4a72-890d-668ebd15c0da\") " pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" Oct 30 21:49:55.276374 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.276011 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrd5\" (UniqueName: \"kubernetes.io/projected/159683c4-0929-4a72-890d-668ebd15c0da-kube-api-access-sdrd5\") pod \"authorino-6f6944bcdd-pr5pp\" (UID: \"159683c4-0929-4a72-890d-668ebd15c0da\") " pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" Oct 30 21:49:55.361597 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.360903 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrd5\" (UniqueName: \"kubernetes.io/projected/159683c4-0929-4a72-890d-668ebd15c0da-kube-api-access-sdrd5\") pod \"authorino-6f6944bcdd-pr5pp\" (UID: \"159683c4-0929-4a72-890d-668ebd15c0da\") " pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" Oct 30 21:49:55.369260 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.368813 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pr5pp"] Oct 30 21:49:55.369260 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.369069 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" Oct 30 21:49:55.397542 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.396276 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58f895975c-xd2qf"] Oct 30 21:49:55.400944 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.400922 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f895975c-xd2qf" Oct 30 21:49:55.479218 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.478300 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85fj\" (UniqueName: \"kubernetes.io/projected/986f6021-37af-4464-897d-d043b2016d0f-kube-api-access-v85fj\") pod \"authorino-58f895975c-xd2qf\" (UID: \"986f6021-37af-4464-897d-d043b2016d0f\") " pod="kuadrant-system/authorino-58f895975c-xd2qf" Oct 30 21:49:55.487355 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.487268 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58f895975c-xd2qf"] Oct 30 21:49:55.580212 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.579902 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85fj\" (UniqueName: \"kubernetes.io/projected/986f6021-37af-4464-897d-d043b2016d0f-kube-api-access-v85fj\") pod \"authorino-58f895975c-xd2qf\" (UID: \"986f6021-37af-4464-897d-d043b2016d0f\") " pod="kuadrant-system/authorino-58f895975c-xd2qf" Oct 30 21:49:55.661973 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.661899 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85fj\" (UniqueName: \"kubernetes.io/projected/986f6021-37af-4464-897d-d043b2016d0f-kube-api-access-v85fj\") pod \"authorino-58f895975c-xd2qf\" (UID: \"986f6021-37af-4464-897d-d043b2016d0f\") " pod="kuadrant-system/authorino-58f895975c-xd2qf" Oct 30 21:49:55.779059 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.778802 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f895975c-xd2qf" Oct 30 21:49:55.860825 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.860762 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pr5pp"] Oct 30 21:49:55.895773 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.895735 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58f895975c-xd2qf"] Oct 30 21:49:55.957620 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.956802 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77cd9bc4d7-pn42f"] Oct 30 21:49:55.965965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:55.965099 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.009204 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.009166 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77cd9bc4d7-pn42f"] Oct 30 21:49:56.019635 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.018922 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 21:49:56.092370 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.092328 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72h4\" (UniqueName: \"kubernetes.io/projected/73d656c4-b4c3-4301-b653-7653675b9aea-kube-api-access-f72h4\") pod \"authorino-77cd9bc4d7-pn42f\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.092915 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.092461 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/73d656c4-b4c3-4301-b653-7653675b9aea-tls-cert\") pod \"authorino-77cd9bc4d7-pn42f\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.193950 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.193858 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/73d656c4-b4c3-4301-b653-7653675b9aea-tls-cert\") pod \"authorino-77cd9bc4d7-pn42f\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.194107 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.193982 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72h4\" (UniqueName: \"kubernetes.io/projected/73d656c4-b4c3-4301-b653-7653675b9aea-kube-api-access-f72h4\") pod \"authorino-77cd9bc4d7-pn42f\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.209694 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.209664 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/73d656c4-b4c3-4301-b653-7653675b9aea-tls-cert\") pod \"authorino-77cd9bc4d7-pn42f\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.238821 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.238780 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58f895975c-xd2qf"] Oct 30 21:49:56.264010 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.263979 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72h4\" (UniqueName: \"kubernetes.io/projected/73d656c4-b4c3-4301-b653-7653675b9aea-kube-api-access-f72h4\") pod \"authorino-77cd9bc4d7-pn42f\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.284843 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.284567 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:49:56.686630 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.686588 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f895975c-xd2qf" event={"ID":"986f6021-37af-4464-897d-d043b2016d0f","Type":"ContainerStarted","Data":"77cb7c543051cc8ad38e40ef94f67998d8214224bee398c09ffabd4f290c1872"} Oct 30 21:49:56.688400 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.688348 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" event={"ID":"159683c4-0929-4a72-890d-668ebd15c0da","Type":"ContainerStarted","Data":"d0691c71db0e3ae37bd2713d1e03affd164c801bc170fe30472502ff3400a9cb"} Oct 30 21:49:56.712746 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:56.712702 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77cd9bc4d7-pn42f"] Oct 30 21:49:56.733754 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:49:56.733726 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73d656c4_b4c3_4301_b653_7653675b9aea.slice/crio-b04f7ff242c0b6536c988c727f4868aee5c183d740fdbe3af0c53dda4a2a5a13 WatchSource:0}: Error finding container b04f7ff242c0b6536c988c727f4868aee5c183d740fdbe3af0c53dda4a2a5a13: Status 404 returned error can't find the container with id b04f7ff242c0b6536c988c727f4868aee5c183d740fdbe3af0c53dda4a2a5a13 Oct 30 21:49:57.700128 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:57.700093 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" event={"ID":"159683c4-0929-4a72-890d-668ebd15c0da","Type":"ContainerStarted","Data":"a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78"} Oct 30 21:49:57.700483 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:57.700259 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" podUID="159683c4-0929-4a72-890d-668ebd15c0da" containerName="authorino" containerID="cri-o://a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78" gracePeriod=30 Oct 30 21:49:57.704188 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:57.703775 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" event={"ID":"387430b1-4ca6-4b40-80a4-6e64323f2f40","Type":"ContainerStarted","Data":"15c6ca7f97de79a001a98d1f11f4c8954593f08621fbdb8116ae6bcecb1ae7f7"} Oct 30 21:49:57.704188 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:57.703859 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:49:57.706789 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:57.706761 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" event={"ID":"73d656c4-b4c3-4301-b653-7653675b9aea","Type":"ContainerStarted","Data":"b04f7ff242c0b6536c988c727f4868aee5c183d740fdbe3af0c53dda4a2a5a13"} Oct 30 21:49:57.818362 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:57.818315 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" podStartSLOduration=2.400595246 podStartE2EDuration="3.81829888s" totalImagesPullingTime="1.417703634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:57.81787075 +0000 UTC m=+446.499020643" watchObservedRunningTime="2026-10-30 21:49:57.81829888 +0000 UTC m=+446.499448772" Oct 30 21:49:57.938582 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:57.938203 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" podStartSLOduration=2.093398796 podStartE2EDuration="5.938184077s" totalImagesPullingTime="3.844785281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:57.935816153 +0000 UTC m=+446.616966073" watchObservedRunningTime="2026-10-30 21:49:57.938184077 +0000 UTC m=+446.619333971" Oct 30 21:49:58.017610 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.017585 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" Oct 30 21:49:58.113243 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.113203 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159683c4-0929-4a72-890d-668ebd15c0da-kube-api-access-sdrd5\" (UniqueName: \"kubernetes.io/projected/159683c4-0929-4a72-890d-668ebd15c0da-kube-api-access-sdrd5\") pod \"159683c4-0929-4a72-890d-668ebd15c0da\" (UID: \"159683c4-0929-4a72-890d-668ebd15c0da\") " Oct 30 21:49:58.116595 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.116442 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159683c4-0929-4a72-890d-668ebd15c0da-kube-api-access-sdrd5" pod "159683c4-0929-4a72-890d-668ebd15c0da" (UID: "159683c4-0929-4a72-890d-668ebd15c0da"). InnerVolumeSpecName "kube-api-access-sdrd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:58.215103 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.215022 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrd5\" (UniqueName: \"kubernetes.io/projected/159683c4-0929-4a72-890d-668ebd15c0da-kube-api-access-sdrd5\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:58.570782 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:49:58.570732 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:49:58.720523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.716724 4933 generic.go:356] "Generic (PLEG): container finished" podID="159683c4-0929-4a72-890d-668ebd15c0da" containerID="a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78" exitCode=0 Oct 30 21:49:58.720523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.716844 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" event={"ID":"159683c4-0929-4a72-890d-668ebd15c0da","Type":"ContainerDied","Data":"a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78"} Oct 30 21:49:58.720523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.716873 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" event={"ID":"159683c4-0929-4a72-890d-668ebd15c0da","Type":"ContainerDied","Data":"d0691c71db0e3ae37bd2713d1e03affd164c801bc170fe30472502ff3400a9cb"} Oct 30 21:49:58.720523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.716888 4933 scope.go:122] "RemoveContainer" containerID="a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78" Oct 30 21:49:58.720523 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.717005 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pr5pp" Oct 30 21:49:58.724637 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.724470 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f895975c-xd2qf" event={"ID":"986f6021-37af-4464-897d-d043b2016d0f","Type":"ContainerStarted","Data":"b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b"} Oct 30 21:49:58.724637 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.724544 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-58f895975c-xd2qf" podUID="986f6021-37af-4464-897d-d043b2016d0f" containerName="authorino" containerID="cri-o://b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b" gracePeriod=30 Oct 30 21:49:58.729556 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.728850 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" event={"ID":"73d656c4-b4c3-4301-b653-7653675b9aea","Type":"ContainerStarted","Data":"0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a"} Oct 30 21:49:58.808893 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.808823 4933 scope.go:122] "RemoveContainer" containerID="a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78" Oct 30 21:49:58.809941 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:49:58.809867 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78\": container with ID starting with a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78 not found: ID does not exist" containerID="a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78" Oct 30 21:49:58.809941 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.809910 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78"} err="failed to get container status \"a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78\": rpc error: code = NotFound desc = could not find container \"a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78\": container with ID starting with a33d386bc76d346a137391ab736bef2da510e94a97bd6b55d7acd24b8df10c78 not found: ID does not exist" Oct 30 21:49:58.890914 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.890105 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pr5pp"] Oct 30 21:49:58.894944 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:58.893117 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pr5pp"] Oct 30 21:49:59.011049 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.010956 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" podStartSLOduration=2.770516799 podStartE2EDuration="4.010914068s" totalImagesPullingTime="1.240397269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:59.008229221 +0000 UTC m=+447.689379115" watchObservedRunningTime="2026-10-30 21:49:59.010914068 +0000 UTC m=+447.692063962" Oct 30 21:49:59.038061 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.037986 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-sphm5"] Oct 30 21:49:59.038311 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.038285 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" podUID="aa6bebf6-77de-4664-84b8-ce0f0a6e8b47" containerName="authorino" containerID="cri-o://11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7" gracePeriod=30 Oct 30 21:49:59.081138 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.078964 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f895975c-xd2qf" Oct 30 21:49:59.127267 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.127165 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-58f895975c-xd2qf" podStartSLOduration=2.704873547 podStartE2EDuration="4.127144962s" totalImagesPullingTime="1.422271415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:59.125748447 +0000 UTC m=+447.806898339" watchObservedRunningTime="2026-10-30 21:49:59.127144962 +0000 UTC m=+447.808294856" Oct 30 21:49:59.227963 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.227933 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986f6021-37af-4464-897d-d043b2016d0f-kube-api-access-v85fj\" (UniqueName: \"kubernetes.io/projected/986f6021-37af-4464-897d-d043b2016d0f-kube-api-access-v85fj\") pod \"986f6021-37af-4464-897d-d043b2016d0f\" (UID: \"986f6021-37af-4464-897d-d043b2016d0f\") " Oct 30 21:49:59.230930 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.230898 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986f6021-37af-4464-897d-d043b2016d0f-kube-api-access-v85fj" pod "986f6021-37af-4464-897d-d043b2016d0f" (UID: "986f6021-37af-4464-897d-d043b2016d0f"). InnerVolumeSpecName "kube-api-access-v85fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:59.300301 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.300275 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" Oct 30 21:49:59.329817 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.329777 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v85fj\" (UniqueName: \"kubernetes.io/projected/986f6021-37af-4464-897d-d043b2016d0f-kube-api-access-v85fj\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.430999 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.430968 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47-kube-api-access-t4fml\" (UniqueName: \"kubernetes.io/projected/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47-kube-api-access-t4fml\") pod \"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47\" (UID: \"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47\") " Oct 30 21:49:59.433430 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.433393 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47-kube-api-access-t4fml" pod "aa6bebf6-77de-4664-84b8-ce0f0a6e8b47" (UID: "aa6bebf6-77de-4664-84b8-ce0f0a6e8b47"). InnerVolumeSpecName "kube-api-access-t4fml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:59.471137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.471109 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159683c4-0929-4a72-890d-668ebd15c0da" path="/var/lib/kubelet/pods/159683c4-0929-4a72-890d-668ebd15c0da/volumes" Oct 30 21:49:59.531697 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.531663 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4fml\" (UniqueName: \"kubernetes.io/projected/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47-kube-api-access-t4fml\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.736970 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.736879 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" event={"ID":"4bb1bfed-1923-4fb4-a7a9-7c34ae56655a","Type":"ContainerStarted","Data":"cbd0ad0aed78a6a8502a22e9f414a15483e32be02d531cabc1c376a65e5fcbc1"} Oct 30 21:49:59.737445 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.737023 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:49:59.738278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.738248 4933 generic.go:356] "Generic (PLEG): container finished" podID="986f6021-37af-4464-897d-d043b2016d0f" containerID="b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b" exitCode=0 Oct 30 21:49:59.738400 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.738349 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58f895975c-xd2qf" Oct 30 21:49:59.738459 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.738392 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f895975c-xd2qf" event={"ID":"986f6021-37af-4464-897d-d043b2016d0f","Type":"ContainerDied","Data":"b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b"} Oct 30 21:49:59.738459 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.738432 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58f895975c-xd2qf" event={"ID":"986f6021-37af-4464-897d-d043b2016d0f","Type":"ContainerDied","Data":"77cb7c543051cc8ad38e40ef94f67998d8214224bee398c09ffabd4f290c1872"} Oct 30 21:49:59.738459 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.738455 4933 scope.go:122] "RemoveContainer" containerID="b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b" Oct 30 21:49:59.741735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.741716 4933 generic.go:356] "Generic (PLEG): container finished" podID="aa6bebf6-77de-4664-84b8-ce0f0a6e8b47" containerID="11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7" exitCode=0 Oct 30 21:49:59.741821 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.741776 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" Oct 30 21:49:59.741885 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.741773 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" event={"ID":"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47","Type":"ContainerDied","Data":"11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7"} Oct 30 21:49:59.741932 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.741886 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-sphm5" event={"ID":"aa6bebf6-77de-4664-84b8-ce0f0a6e8b47","Type":"ContainerDied","Data":"9f54d15e08cf878d45e2e667729f1987ccd7ce88c11d8238c99017be67e02eed"} Oct 30 21:49:59.743830 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.743807 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" event={"ID":"b1c9c99e-1ac0-4a2c-b722-0acc313ced68","Type":"ContainerStarted","Data":"80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663"} Oct 30 21:49:59.744009 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.743995 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:49:59.749546 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.749478 4933 scope.go:122] "RemoveContainer" containerID="b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b" Oct 30 21:49:59.749844 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:49:59.749820 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b\": container with ID starting with b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b not found: ID does not exist" containerID="b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b" Oct 30 21:49:59.749924 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.749857 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b"} err="failed to get container status \"b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b\": rpc error: code = NotFound desc = could not find container \"b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b\": container with ID starting with b87c421934f39cd74794081f6b48d5da65a9211f67f07ab9aeaed24e239c8e4b not found: ID does not exist" Oct 30 21:49:59.749924 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.749880 4933 scope.go:122] "RemoveContainer" containerID="11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7" Oct 30 21:49:59.759870 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.759856 4933 scope.go:122] "RemoveContainer" containerID="11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7" Oct 30 21:49:59.760152 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:49:59.760131 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7\": container with ID starting with 11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7 not found: ID does not exist" containerID="11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7" Oct 30 21:49:59.760204 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.760164 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7"} err="failed to get container status \"11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7\": rpc error: code = NotFound desc = could not find container \"11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7\": container with ID starting with 11deb2d0a618fb1b85fec83eca52de36aea9ad9bcdbcdeffad27e8be7a4a7ea7 not found: ID does not exist" Oct 30 21:49:59.861533 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.861461 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" podStartSLOduration=2.51683223 podStartE2EDuration="7.861447642s" totalImagesPullingTime="5.344615412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:59.857797714 +0000 UTC m=+448.538947607" watchObservedRunningTime="2026-10-30 21:49:59.861447642 +0000 UTC m=+448.542597534" Oct 30 21:49:59.970308 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.970274 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58f895975c-xd2qf"] Oct 30 21:49:59.975191 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:49:59.975165 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58f895975c-xd2qf"] Oct 30 21:50:00.088932 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.088902 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-sphm5"] Oct 30 21:50:00.093587 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.093560 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-sphm5"] Oct 30 21:50:00.212571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.212515 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" podStartSLOduration=2.745299885 podStartE2EDuration="8.212502509s" totalImagesPullingTime="5.467202624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:00.211032917 +0000 UTC m=+448.892182818" watchObservedRunningTime="2026-10-30 21:50:00.212502509 +0000 UTC m=+448.893652393" Oct 30 21:50:00.751593 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.751514 4933 generic.go:356] "Generic (PLEG): container finished" podID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerID="93bbb939a0c8ea9833e5b3afc567cec2fe76a7dbeaaa15ae52dfc6338a8d128e" exitCode=0 Oct 30 21:50:00.751593 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.751570 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" event={"ID":"acef2b25-5c28-44db-8c2b-ba32a12ddce6","Type":"ContainerDied","Data":"93bbb939a0c8ea9833e5b3afc567cec2fe76a7dbeaaa15ae52dfc6338a8d128e"} Oct 30 21:50:00.752100 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.751916 4933 scope.go:122] "RemoveContainer" containerID="93bbb939a0c8ea9833e5b3afc567cec2fe76a7dbeaaa15ae52dfc6338a8d128e" Oct 30 21:50:00.752976 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.752933 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:00.753060 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.753041 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:00.753111 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:00.753095 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:50:01.468712 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:01.468683 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986f6021-37af-4464-897d-d043b2016d0f" path="/var/lib/kubelet/pods/986f6021-37af-4464-897d-d043b2016d0f/volumes" Oct 30 21:50:01.469049 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:01.469035 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6bebf6-77de-4664-84b8-ce0f0a6e8b47" path="/var/lib/kubelet/pods/aa6bebf6-77de-4664-84b8-ce0f0a6e8b47/volumes" Oct 30 21:50:01.761267 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:01.761183 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" event={"ID":"acef2b25-5c28-44db-8c2b-ba32a12ddce6","Type":"ContainerStarted","Data":"c767dcdd54e37b18ac465d45c89d2ff7cb9a2455e613e167dfec83350e88172d"} Oct 30 21:50:01.761822 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:01.761807 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:50:01.764650 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:01.764627 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:50:01.764781 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:01.764677 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:50:02.762080 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:02.762043 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:50:02.762470 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:02.762097 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:50:03.762007 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:03.761972 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:50:03.762209 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:03.762029 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:50:04.762724 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:04.762692 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:50:04.763129 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:04.762740 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:50:05.763008 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:05.762974 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:50:05.763522 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:05.763046 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:50:06.780312 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:06.780282 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:50:06.780810 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:06.780708 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:50:06.781580 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:06.781562 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" Oct 30 21:50:08.731933 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:08.731909 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-jpwxz" Oct 30 21:50:28.543169 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:50:28.543135 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:50:30.755022 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:30.754999 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:50:30.758111 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:50:30.758092 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-86z25" Oct 30 21:50:58.541728 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:50:58.541644 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:51:00.444135 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444105 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ntncq"] Oct 30 21:51:00.444571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444517 4933 cpu_manager.go:414] "containerMap: removing container" podUID="986f6021-37af-4464-897d-d043b2016d0f" containerName="authorino" Oct 30 21:51:00.444571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444532 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986f6021-37af-4464-897d-d043b2016d0f" containerName="authorino" Oct 30 21:51:00.444571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444558 4933 cpu_manager.go:414] "containerMap: removing container" podUID="aa6bebf6-77de-4664-84b8-ce0f0a6e8b47" containerName="authorino" Oct 30 21:51:00.444571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444564 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6bebf6-77de-4664-84b8-ce0f0a6e8b47" containerName="authorino" Oct 30 21:51:00.444715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444581 4933 cpu_manager.go:414] "containerMap: removing container" podUID="159683c4-0929-4a72-890d-668ebd15c0da" containerName="authorino" Oct 30 21:51:00.444715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444586 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159683c4-0929-4a72-890d-668ebd15c0da" containerName="authorino" Oct 30 21:51:00.444715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444681 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="986f6021-37af-4464-897d-d043b2016d0f" containerName="authorino" Oct 30 21:51:00.444715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444691 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6bebf6-77de-4664-84b8-ce0f0a6e8b47" containerName="authorino" Oct 30 21:51:00.444715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.444700 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="159683c4-0929-4a72-890d-668ebd15c0da" containerName="authorino" Oct 30 21:51:00.447030 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.447016 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:00.494798 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.494773 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxhp\" (UniqueName: \"kubernetes.io/projected/c4a51477-e289-421a-8a6e-3d720975fdc2-kube-api-access-bzxhp\") pod \"llmisvc-controller-manager-6466c554d8-ntncq\" (UID: \"c4a51477-e289-421a-8a6e-3d720975fdc2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:00.494928 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.494827 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4a51477-e289-421a-8a6e-3d720975fdc2-cert\") pod \"llmisvc-controller-manager-6466c554d8-ntncq\" (UID: \"c4a51477-e289-421a-8a6e-3d720975fdc2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:00.504917 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.504893 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ntncq"] Oct 30 21:51:00.595179 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.595147 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxhp\" (UniqueName: \"kubernetes.io/projected/c4a51477-e289-421a-8a6e-3d720975fdc2-kube-api-access-bzxhp\") pod \"llmisvc-controller-manager-6466c554d8-ntncq\" (UID: \"c4a51477-e289-421a-8a6e-3d720975fdc2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:00.595363 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.595223 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4a51477-e289-421a-8a6e-3d720975fdc2-cert\") pod \"llmisvc-controller-manager-6466c554d8-ntncq\" (UID: \"c4a51477-e289-421a-8a6e-3d720975fdc2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:00.597831 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.597806 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c4a51477-e289-421a-8a6e-3d720975fdc2-cert\") pod \"llmisvc-controller-manager-6466c554d8-ntncq\" (UID: \"c4a51477-e289-421a-8a6e-3d720975fdc2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:00.653980 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.653945 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxhp\" (UniqueName: \"kubernetes.io/projected/c4a51477-e289-421a-8a6e-3d720975fdc2-kube-api-access-bzxhp\") pod \"llmisvc-controller-manager-6466c554d8-ntncq\" (UID: \"c4a51477-e289-421a-8a6e-3d720975fdc2\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:00.758878 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:00.758815 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:01.130331 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:01.130304 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-ntncq"] Oct 30 21:51:01.131420 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:51:01.131393 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a51477_e289_421a_8a6e_3d720975fdc2.slice/crio-fc70ba136499c193da8b2f4dc9e0f0b2725a7a06e642d9c42ce76e3f6e279854 WatchSource:0}: Error finding container fc70ba136499c193da8b2f4dc9e0f0b2725a7a06e642d9c42ce76e3f6e279854: Status 404 returned error can't find the container with id fc70ba136499c193da8b2f4dc9e0f0b2725a7a06e642d9c42ce76e3f6e279854 Oct 30 21:51:02.078673 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:02.078645 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" event={"ID":"c4a51477-e289-421a-8a6e-3d720975fdc2","Type":"ContainerStarted","Data":"1a506b7a805d7eec563fbe11718313e65bc605d20bedfd88ad218bb749384f6a"} Oct 30 21:51:02.079028 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:02.078689 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" event={"ID":"c4a51477-e289-421a-8a6e-3d720975fdc2","Type":"ContainerStarted","Data":"fc70ba136499c193da8b2f4dc9e0f0b2725a7a06e642d9c42ce76e3f6e279854"} Oct 30 21:51:02.079028 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:02.078737 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:02.194774 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:02.194686 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" podStartSLOduration=1.4359535939999999 podStartE2EDuration="2.194674664s" totalImagesPullingTime="758.72107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:02.194023783 +0000 UTC m=+510.875173676" watchObservedRunningTime="2026-10-30 21:51:02.194674664 +0000 UTC m=+510.875824557" Oct 30 21:51:28.543366 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:51:28.543332 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:51:33.085744 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.085710 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-ntncq" Oct 30 21:51:33.257715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.257686 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl"] Oct 30 21:51:33.257966 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.257934 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" podUID="b1c9c99e-1ac0-4a2c-b722-0acc313ced68" containerName="manager" containerID="cri-o://80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663" gracePeriod=30 Oct 30 21:51:33.497333 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.497306 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:51:33.646097 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.646036 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-cert\" (UniqueName: \"kubernetes.io/secret/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-cert\") pod \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " Oct 30 21:51:33.646097 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.646081 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-kube-api-access-6k5ct\" (UniqueName: \"kubernetes.io/projected/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-kube-api-access-6k5ct\") pod \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\" (UID: \"b1c9c99e-1ac0-4a2c-b722-0acc313ced68\") " Oct 30 21:51:33.648361 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.648338 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-cert" pod "b1c9c99e-1ac0-4a2c-b722-0acc313ced68" (UID: "b1c9c99e-1ac0-4a2c-b722-0acc313ced68"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:33.648474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.648402 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-kube-api-access-6k5ct" pod "b1c9c99e-1ac0-4a2c-b722-0acc313ced68" (UID: "b1c9c99e-1ac0-4a2c-b722-0acc313ced68"). InnerVolumeSpecName "kube-api-access-6k5ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:33.747190 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.747166 4933 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:33.747190 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:33.747189 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k5ct\" (UniqueName: \"kubernetes.io/projected/b1c9c99e-1ac0-4a2c-b722-0acc313ced68-kube-api-access-6k5ct\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:51:34.243224 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.243198 4933 generic.go:356] "Generic (PLEG): container finished" podID="b1c9c99e-1ac0-4a2c-b722-0acc313ced68" containerID="80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663" exitCode=0 Oct 30 21:51:34.243721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.243239 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" event={"ID":"b1c9c99e-1ac0-4a2c-b722-0acc313ced68","Type":"ContainerDied","Data":"80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663"} Oct 30 21:51:34.243721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.243244 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" Oct 30 21:51:34.243721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.243274 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl" event={"ID":"b1c9c99e-1ac0-4a2c-b722-0acc313ced68","Type":"ContainerDied","Data":"a0c5e095bf735bcf025a704d1b48bed2dbdc20daffcd9cb6dd41b1936730a328"} Oct 30 21:51:34.243721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.243296 4933 scope.go:122] "RemoveContainer" containerID="80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663" Oct 30 21:51:34.253198 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.253182 4933 scope.go:122] "RemoveContainer" containerID="80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663" Oct 30 21:51:34.253453 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:51:34.253435 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663\": container with ID starting with 80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663 not found: ID does not exist" containerID="80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663" Oct 30 21:51:34.253625 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.253460 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663"} err="failed to get container status \"80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663\": rpc error: code = NotFound desc = could not find container \"80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663\": container with ID starting with 80cf6f9b027756c78bb5421dc935ec41da1ebe0496e23b162eba081007ecb663 not found: ID does not exist" Oct 30 21:51:34.414424 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.414400 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl"] Oct 30 21:51:34.464976 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:34.464957 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-56rkl"] Oct 30 21:51:35.463661 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:35.463623 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c9c99e-1ac0-4a2c-b722-0acc313ced68" path="/var/lib/kubelet/pods/b1c9c99e-1ac0-4a2c-b722-0acc313ced68/volumes" Oct 30 21:51:56.784151 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:51:56.784115 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:51:58.542543 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:51:58.542508 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:52:06.783233 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:06.783205 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:52:20.478549 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:20.478449 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:52:28.548821 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:52:28.548785 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:52:31.538177 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:31.538145 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:52:31.541041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:31.541012 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:52:33.837393 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:33.837359 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:52:43.402159 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.402120 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-676b586c44-xvzr8"] Oct 30 21:52:43.402745 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.402725 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b1c9c99e-1ac0-4a2c-b722-0acc313ced68" containerName="manager" Oct 30 21:52:43.402827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.402748 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c9c99e-1ac0-4a2c-b722-0acc313ced68" containerName="manager" Oct 30 21:52:43.402886 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.402858 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c9c99e-1ac0-4a2c-b722-0acc313ced68" containerName="manager" Oct 30 21:52:43.405797 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.405777 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:43.474418 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.474385 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-676b586c44-xvzr8"] Oct 30 21:52:43.529376 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.529326 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91184aa4-0bba-49fb-a9a7-c3fca31ce6e7-tls-cert\") pod \"authorino-676b586c44-xvzr8\" (UID: \"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7\") " pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:43.529546 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.529521 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz9nj\" (UniqueName: \"kubernetes.io/projected/91184aa4-0bba-49fb-a9a7-c3fca31ce6e7-kube-api-access-dz9nj\") pod \"authorino-676b586c44-xvzr8\" (UID: \"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7\") " pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:43.630060 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.630025 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz9nj\" (UniqueName: \"kubernetes.io/projected/91184aa4-0bba-49fb-a9a7-c3fca31ce6e7-kube-api-access-dz9nj\") pod \"authorino-676b586c44-xvzr8\" (UID: \"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7\") " pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:43.630211 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.630090 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91184aa4-0bba-49fb-a9a7-c3fca31ce6e7-tls-cert\") pod \"authorino-676b586c44-xvzr8\" (UID: \"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7\") " pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:43.632750 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.632723 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91184aa4-0bba-49fb-a9a7-c3fca31ce6e7-tls-cert\") pod \"authorino-676b586c44-xvzr8\" (UID: \"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7\") " pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:43.691383 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.691312 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz9nj\" (UniqueName: \"kubernetes.io/projected/91184aa4-0bba-49fb-a9a7-c3fca31ce6e7-kube-api-access-dz9nj\") pod \"authorino-676b586c44-xvzr8\" (UID: \"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7\") " pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:43.717184 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:43.717156 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-676b586c44-xvzr8" Oct 30 21:52:44.090147 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:44.090116 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-676b586c44-xvzr8"] Oct 30 21:52:44.091616 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:52:44.091575 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91184aa4_0bba_49fb_a9a7_c3fca31ce6e7.slice/crio-358b594f40eeb2edf0a544413512befb7e1d5f3618f4a4b57defbd653f03e19f WatchSource:0}: Error finding container 358b594f40eeb2edf0a544413512befb7e1d5f3618f4a4b57defbd653f03e19f: Status 404 returned error can't find the container with id 358b594f40eeb2edf0a544413512befb7e1d5f3618f4a4b57defbd653f03e19f Oct 30 21:52:44.093029 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:44.093010 4933 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:52:44.625284 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:44.625252 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-676b586c44-xvzr8" event={"ID":"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7","Type":"ContainerStarted","Data":"358b594f40eeb2edf0a544413512befb7e1d5f3618f4a4b57defbd653f03e19f"} Oct 30 21:52:45.631531 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:45.631468 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-676b586c44-xvzr8" event={"ID":"91184aa4-0bba-49fb-a9a7-c3fca31ce6e7","Type":"ContainerStarted","Data":"3a62316a7ac8d4702ee52540af81fc6a2c9c8600d01917ecb6200e6134f05642"} Oct 30 21:52:45.746914 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:45.746823 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-676b586c44-xvzr8" podStartSLOduration=1.366943407 podStartE2EDuration="2.746808989s" totalImagesPullingTime="1.379865582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:45.745409676 +0000 UTC m=+614.426559569" watchObservedRunningTime="2026-10-30 21:52:45.746808989 +0000 UTC m=+614.427958921" Oct 30 21:52:45.769380 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:45.769355 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77cd9bc4d7-pn42f"] Oct 30 21:52:45.769595 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:45.769575 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" podUID="73d656c4-b4c3-4301-b653-7653675b9aea" containerName="authorino" containerID="cri-o://0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a" gracePeriod=30 Oct 30 21:52:46.023643 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.023622 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:52:46.153589 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.153560 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d656c4-b4c3-4301-b653-7653675b9aea-tls-cert\" (UniqueName: \"kubernetes.io/secret/73d656c4-b4c3-4301-b653-7653675b9aea-tls-cert\") pod \"73d656c4-b4c3-4301-b653-7653675b9aea\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " Oct 30 21:52:46.153788 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.153609 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d656c4-b4c3-4301-b653-7653675b9aea-kube-api-access-f72h4\" (UniqueName: \"kubernetes.io/projected/73d656c4-b4c3-4301-b653-7653675b9aea-kube-api-access-f72h4\") pod \"73d656c4-b4c3-4301-b653-7653675b9aea\" (UID: \"73d656c4-b4c3-4301-b653-7653675b9aea\") " Oct 30 21:52:46.156617 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.156570 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d656c4-b4c3-4301-b653-7653675b9aea-kube-api-access-f72h4" pod "73d656c4-b4c3-4301-b653-7653675b9aea" (UID: "73d656c4-b4c3-4301-b653-7653675b9aea"). InnerVolumeSpecName "kube-api-access-f72h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:46.165995 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.165968 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d656c4-b4c3-4301-b653-7653675b9aea-tls-cert" pod "73d656c4-b4c3-4301-b653-7653675b9aea" (UID: "73d656c4-b4c3-4301-b653-7653675b9aea"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:46.255312 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.255284 4933 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/73d656c4-b4c3-4301-b653-7653675b9aea-tls-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:46.255312 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.255312 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f72h4\" (UniqueName: \"kubernetes.io/projected/73d656c4-b4c3-4301-b653-7653675b9aea-kube-api-access-f72h4\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:52:46.641929 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.641892 4933 generic.go:356] "Generic (PLEG): container finished" podID="73d656c4-b4c3-4301-b653-7653675b9aea" containerID="0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a" exitCode=0 Oct 30 21:52:46.642405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.641935 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" Oct 30 21:52:46.642405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.641972 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" event={"ID":"73d656c4-b4c3-4301-b653-7653675b9aea","Type":"ContainerDied","Data":"0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a"} Oct 30 21:52:46.642405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.642009 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cd9bc4d7-pn42f" event={"ID":"73d656c4-b4c3-4301-b653-7653675b9aea","Type":"ContainerDied","Data":"b04f7ff242c0b6536c988c727f4868aee5c183d740fdbe3af0c53dda4a2a5a13"} Oct 30 21:52:46.642405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.642028 4933 scope.go:122] "RemoveContainer" containerID="0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a" Oct 30 21:52:46.652798 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.652781 4933 scope.go:122] "RemoveContainer" containerID="0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a" Oct 30 21:52:46.653068 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:52:46.653050 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a\": container with ID starting with 0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a not found: ID does not exist" containerID="0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a" Oct 30 21:52:46.653135 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.653079 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a"} err="failed to get container status \"0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a\": rpc error: code = NotFound desc = could not find container \"0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a\": container with ID starting with 0c6dcedc0359d3c361334f961969b6a7b8ba76914a86d881dd4d2620ad3b832a not found: ID does not exist" Oct 30 21:52:46.819538 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.819507 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77cd9bc4d7-pn42f"] Oct 30 21:52:46.835324 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:46.835291 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77cd9bc4d7-pn42f"] Oct 30 21:52:47.465267 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:52:47.465227 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d656c4-b4c3-4301-b653-7653675b9aea" path="/var/lib/kubelet/pods/73d656c4-b4c3-4301-b653-7653675b9aea/volumes" Oct 30 21:52:58.543333 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:52:58.543287 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:53:28.543243 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:53:28.543208 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:53:58.543891 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:53:58.543809 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:54:06.569976 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.569937 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm"] Oct 30 21:54:06.570436 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.570350 4933 cpu_manager.go:414] "containerMap: removing container" podUID="73d656c4-b4c3-4301-b653-7653675b9aea" containerName="authorino" Oct 30 21:54:06.570436 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.570361 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d656c4-b4c3-4301-b653-7653675b9aea" containerName="authorino" Oct 30 21:54:06.570436 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.570425 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d656c4-b4c3-4301-b653-7653675b9aea" containerName="authorino" Oct 30 21:54:06.573722 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.573706 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:06.628848 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.628820 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-9f8aea\"" Oct 30 21:54:06.679280 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.679249 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-9f8aea-dockercfg-qbpvx\"" Oct 30 21:54:06.681405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.681379 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm"] Oct 30 21:54:06.689819 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.689790 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs55h\" (UniqueName: \"kubernetes.io/projected/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-kube-api-access-rs55h\") pod \"payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:06.689955 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.689931 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:06.790985 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.790962 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs55h\" (UniqueName: \"kubernetes.io/projected/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-kube-api-access-rs55h\") pod \"payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:06.791116 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.791020 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:06.791673 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.791656 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:06.849915 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.849858 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs55h\" (UniqueName: \"kubernetes.io/projected/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-kube-api-access-rs55h\") pod \"payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:06.886731 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:06.886708 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:07.269251 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:07.269153 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm"] Oct 30 21:54:07.272569 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:54:07.272536 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c28fcf_5e62_4b4d_ae03_c7e16b464bc7.slice/crio-9dfc3b362c6a450bfabea2ad6e413c198a35e43fc25d85cd75ae06bea8459a1d WatchSource:0}: Error finding container 9dfc3b362c6a450bfabea2ad6e413c198a35e43fc25d85cd75ae06bea8459a1d: Status 404 returned error can't find the container with id 9dfc3b362c6a450bfabea2ad6e413c198a35e43fc25d85cd75ae06bea8459a1d Oct 30 21:54:08.067839 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:08.067795 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" event={"ID":"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7","Type":"ContainerStarted","Data":"9dfc3b362c6a450bfabea2ad6e413c198a35e43fc25d85cd75ae06bea8459a1d"} Oct 30 21:54:08.764639 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:08.764597 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:54:08.931852 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:08.931819 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn"] Oct 30 21:54:08.935663 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:08.935637 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:08.993187 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:08.993155 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn"] Oct 30 21:54:09.012671 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.012632 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f675717-5864-48b6-a2d3-88605c5558c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:09.012882 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.012722 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtdv\" (UniqueName: \"kubernetes.io/projected/7f675717-5864-48b6-a2d3-88605c5558c7-kube-api-access-zrtdv\") pod \"payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:09.041134 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.041099 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d516c4-dockercfg-fzchv\"" Oct 30 21:54:09.041134 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.041124 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d516c4\"" Oct 30 21:54:09.113261 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.113218 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtdv\" (UniqueName: \"kubernetes.io/projected/7f675717-5864-48b6-a2d3-88605c5558c7-kube-api-access-zrtdv\") pod \"payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:09.113825 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.113300 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f675717-5864-48b6-a2d3-88605c5558c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:09.114247 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.114221 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f675717-5864-48b6-a2d3-88605c5558c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:09.179562 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.179526 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtdv\" (UniqueName: \"kubernetes.io/projected/7f675717-5864-48b6-a2d3-88605c5558c7-kube-api-access-zrtdv\") pod \"payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:09.253633 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.253598 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:09.643375 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:09.643341 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn"] Oct 30 21:54:09.645567 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:54:09.645484 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f675717_5864_48b6_a2d3_88605c5558c7.slice/crio-f6e35b2a768ca76aca3572427e079651070a01b387cfd80fbca854fa3c9901a6 WatchSource:0}: Error finding container f6e35b2a768ca76aca3572427e079651070a01b387cfd80fbca854fa3c9901a6: Status 404 returned error can't find the container with id f6e35b2a768ca76aca3572427e079651070a01b387cfd80fbca854fa3c9901a6 Oct 30 21:54:10.080656 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:10.080628 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" event={"ID":"7f675717-5864-48b6-a2d3-88605c5558c7","Type":"ContainerStarted","Data":"f6e35b2a768ca76aca3572427e079651070a01b387cfd80fbca854fa3c9901a6"} Oct 30 21:54:10.082307 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:10.082284 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" event={"ID":"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7","Type":"ContainerStarted","Data":"08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8"} Oct 30 21:54:10.082438 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:10.082423 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:10.201705 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:10.201656 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" podStartSLOduration=1.472164737 podStartE2EDuration="4.20164293s" totalImagesPullingTime="2.729478193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.200628157 +0000 UTC m=+698.881778064" watchObservedRunningTime="2026-10-30 21:54:10.20164293 +0000 UTC m=+698.882792822" Oct 30 21:54:11.089338 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.089301 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" event={"ID":"7f675717-5864-48b6-a2d3-88605c5558c7","Type":"ContainerStarted","Data":"9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66"} Oct 30 21:54:11.089557 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.089375 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:11.211929 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.211880 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" podStartSLOduration=2.256844854 podStartE2EDuration="3.211865996s" totalImagesPullingTime="955.021142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:11.211683802 +0000 UTC m=+699.892833694" watchObservedRunningTime="2026-10-30 21:54:11.211865996 +0000 UTC m=+699.893015889" Oct 30 21:54:11.426143 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.426068 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm"] Oct 30 21:54:11.429639 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.429624 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:11.433937 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.433899 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:11.434081 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.434064 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7j7\" (UniqueName: \"kubernetes.io/projected/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-kube-api-access-xw7j7\") pod \"payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:11.485920 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.485895 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-29cbad-dockercfg-449h4\"" Oct 30 21:54:11.492500 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.492469 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm"] Oct 30 21:54:11.534645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.534607 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:11.534840 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.534745 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7j7\" (UniqueName: \"kubernetes.io/projected/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-kube-api-access-xw7j7\") pod \"payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:11.571746 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.571720 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-29cbad\"" Oct 30 21:54:11.575397 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.575375 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:11.595745 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.595714 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7j7\" (UniqueName: \"kubernetes.io/projected/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-kube-api-access-xw7j7\") pod \"payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:11.742240 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:11.742143 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:12.121835 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:12.121802 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm"] Oct 30 21:54:12.123192 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:54:12.123161 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e696cd7_f7b8_4da0_bf63_00b0ecd3c266.slice/crio-45c9cdd2472aa234ea14219ea22a469c5d0eb806d6165e6c6442608940834617 WatchSource:0}: Error finding container 45c9cdd2472aa234ea14219ea22a469c5d0eb806d6165e6c6442608940834617: Status 404 returned error can't find the container with id 45c9cdd2472aa234ea14219ea22a469c5d0eb806d6165e6c6442608940834617 Oct 30 21:54:13.100941 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:13.100904 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" event={"ID":"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266","Type":"ContainerStarted","Data":"7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f"} Oct 30 21:54:13.101409 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:13.100962 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" event={"ID":"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266","Type":"ContainerStarted","Data":"45c9cdd2472aa234ea14219ea22a469c5d0eb806d6165e6c6442608940834617"} Oct 30 21:54:13.101409 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:13.100987 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:13.222634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:13.222570 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" podStartSLOduration=2.222553008 podStartE2EDuration="2.222553008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:13.220969934 +0000 UTC m=+701.902119831" watchObservedRunningTime="2026-10-30 21:54:13.222553008 +0000 UTC m=+701.903702898" Oct 30 21:54:14.198513 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:14.186049 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:54:21.092472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:21.092442 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 21:54:22.097227 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:22.097198 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 21:54:24.108816 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:24.108790 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 21:54:28.542638 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:54:28.542605 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:54:29.477258 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:29.477226 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:54:44.992959 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:44.992924 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf"] Oct 30 21:54:45.003504 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.003466 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.056524 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.056449 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf"] Oct 30 21:54:45.103775 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.103747 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"openshift-service-ca.crt\"" Oct 30 21:54:45.103775 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.103772 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"simulator-w4-b0c55f-kserve-self-signed-certs\"" Oct 30 21:54:45.103958 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.103777 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"kube-root-ca.crt\"" Oct 30 21:54:45.108721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.108702 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"default-dockercfg-sdlj2\"" Oct 30 21:54:45.124764 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.124735 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11940b-1bc9-4503-8788-00f9e8000787-tls-certs\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.124879 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.124786 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-dshm\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.124879 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.124827 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-tmp-dir\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.124879 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.124861 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-model-cache\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.125011 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.124947 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-home\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.125011 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.124992 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf2bq\" (UniqueName: \"kubernetes.io/projected/2a11940b-1bc9-4503-8788-00f9e8000787-kube-api-access-hf2bq\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.225390 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225364 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11940b-1bc9-4503-8788-00f9e8000787-tls-certs\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.225567 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225410 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-dshm\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.225567 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225539 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-tmp-dir\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.225662 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225575 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-model-cache\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.225662 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225642 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-home\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.225762 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225666 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf2bq\" (UniqueName: \"kubernetes.io/projected/2a11940b-1bc9-4503-8788-00f9e8000787-kube-api-access-hf2bq\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.225929 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225900 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-tmp-dir\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.226135 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.225971 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-model-cache\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.226135 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.226019 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-home\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.227733 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.227713 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-dshm\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.228074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.228057 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11940b-1bc9-4503-8788-00f9e8000787-tls-certs\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.283265 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.283241 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf2bq\" (UniqueName: \"kubernetes.io/projected/2a11940b-1bc9-4503-8788-00f9e8000787-kube-api-access-hf2bq\") pod \"simulator-w4-b0c55f-kserve-58c999cffb-jjzcf\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.317974 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.317950 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:45.692977 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:45.692954 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf"] Oct 30 21:54:45.694403 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:54:45.694372 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a11940b_1bc9_4503_8788_00f9e8000787.slice/crio-60cd0529829ea0a21f3c991bcff23d3356d73d26c3477b9963fccb9ad137e042 WatchSource:0}: Error finding container 60cd0529829ea0a21f3c991bcff23d3356d73d26c3477b9963fccb9ad137e042: Status 404 returned error can't find the container with id 60cd0529829ea0a21f3c991bcff23d3356d73d26c3477b9963fccb9ad137e042 Oct 30 21:54:46.293519 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:46.293457 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" event={"ID":"2a11940b-1bc9-4503-8788-00f9e8000787","Type":"ContainerStarted","Data":"60cd0529829ea0a21f3c991bcff23d3356d73d26c3477b9963fccb9ad137e042"} Oct 30 21:54:47.988514 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:47.967321 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:54:48.308174 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:48.308144 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" event={"ID":"2a11940b-1bc9-4503-8788-00f9e8000787","Type":"ContainerStarted","Data":"9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f"} Oct 30 21:54:48.318426 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:48.318409 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:48.422394 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:48.422340 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podStartSLOduration=2.7463422509999997 podStartE2EDuration="4.422319752s" totalImagesPullingTime="1.675977501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:48.421658581 +0000 UTC m=+737.102808473" watchObservedRunningTime="2026-10-30 21:54:48.422319752 +0000 UTC m=+737.103469646" Oct 30 21:54:55.318175 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:55.318143 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:55.330880 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:55.330856 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:55.356948 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:55.356926 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 21:54:58.465843 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:54:58.465802 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:54:58.544981 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:54:58.544947 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:55:08.464119 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.464085 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb"] Oct 30 21:55:08.467606 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.467592 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.548347 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.548310 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb"] Oct 30 21:55:08.574553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.574519 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"simulator-w5-9f8aea-kserve-self-signed-certs\"" Oct 30 21:55:08.574553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.574520 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"default-dockercfg-vm2j7\"" Oct 30 21:55:08.574721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.574594 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"kube-root-ca.crt\"" Oct 30 21:55:08.574766 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.574733 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"openshift-service-ca.crt\"" Oct 30 21:55:08.627597 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.627571 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c201be91-d35d-4319-9fea-01c061905c27-tls-certs\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.627786 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.627603 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-home\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.627786 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.627637 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-dshm\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.627786 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.627658 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-tmp-dir\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.627949 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.627798 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmsq\" (UniqueName: \"kubernetes.io/projected/c201be91-d35d-4319-9fea-01c061905c27-kube-api-access-jrmsq\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.627949 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.627855 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-model-cache\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.728952 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.728874 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-dshm\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.728952 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.728916 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-tmp-dir\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.729458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.728966 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmsq\" (UniqueName: \"kubernetes.io/projected/c201be91-d35d-4319-9fea-01c061905c27-kube-api-access-jrmsq\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.729458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.729034 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-model-cache\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.729458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.729239 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c201be91-d35d-4319-9fea-01c061905c27-tls-certs\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.729458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.729344 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-tmp-dir\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.729458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.729416 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-model-cache\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.729778 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.729717 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-home\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.730053 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.730024 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-home\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.731627 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.731602 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-dshm\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.732074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.732051 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c201be91-d35d-4319-9fea-01c061905c27-tls-certs\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:08.794827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:08.794794 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmsq\" (UniqueName: \"kubernetes.io/projected/c201be91-d35d-4319-9fea-01c061905c27-kube-api-access-jrmsq\") pod \"simulator-w5-9f8aea-kserve-56ddfcb466-scdxb\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:09.079451 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:09.079406 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:09.455189 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:09.455159 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb"] Oct 30 21:55:09.456034 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:55:09.456011 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc201be91_d35d_4319_9fea_01c061905c27.slice/crio-040e7880d4d7803c0c967d03ca0f7c3c28eb25faab36d1e7b4df967707aaf97f WatchSource:0}: Error finding container 040e7880d4d7803c0c967d03ca0f7c3c28eb25faab36d1e7b4df967707aaf97f: Status 404 returned error can't find the container with id 040e7880d4d7803c0c967d03ca0f7c3c28eb25faab36d1e7b4df967707aaf97f Oct 30 21:55:10.438652 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:10.438616 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" event={"ID":"c201be91-d35d-4319-9fea-01c061905c27","Type":"ContainerStarted","Data":"d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3"} Oct 30 21:55:10.439126 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:10.438667 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" event={"ID":"c201be91-d35d-4319-9fea-01c061905c27","Type":"ContainerStarted","Data":"040e7880d4d7803c0c967d03ca0f7c3c28eb25faab36d1e7b4df967707aaf97f"} Oct 30 21:55:10.555579 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:10.555466 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podStartSLOduration=2.555454387 podStartE2EDuration="2.555454387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:10.554530365 +0000 UTC m=+759.235680258" watchObservedRunningTime="2026-10-30 21:55:10.555454387 +0000 UTC m=+759.236604279" Oct 30 21:55:11.079749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:11.079716 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:14.000616 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:14.000580 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:55:19.079925 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:19.079896 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:19.092599 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:19.092573 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:19.499983 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:19.499895 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 21:55:26.181894 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.181853 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9"] Oct 30 21:55:26.188180 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.188157 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.258740 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.258701 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"kube-root-ca.crt\"" Oct 30 21:55:26.258898 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.258740 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"openshift-service-ca.crt\"" Oct 30 21:55:26.261075 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.261052 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9"] Oct 30 21:55:26.285271 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.285248 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-home\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.285370 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.285278 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-tmp-dir\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.285418 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.285387 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-model-cache\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.285462 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.285423 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef2246f-256c-4851-81e9-b8dbf577ff54-tls-certs\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.285542 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.285521 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-dshm\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.285592 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.285561 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwdg\" (UniqueName: \"kubernetes.io/projected/9ef2246f-256c-4851-81e9-b8dbf577ff54-kube-api-access-dfwdg\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.293261 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.293236 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"simulator-w1-d516c4-kserve-self-signed-certs\"" Oct 30 21:55:26.293261 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.293242 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"default-dockercfg-t65jc\"" Oct 30 21:55:26.386226 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386192 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-model-cache\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386396 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386240 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef2246f-256c-4851-81e9-b8dbf577ff54-tls-certs\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386396 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386274 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-dshm\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386396 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386325 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwdg\" (UniqueName: \"kubernetes.io/projected/9ef2246f-256c-4851-81e9-b8dbf577ff54-kube-api-access-dfwdg\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386396 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386359 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-home\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386396 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386386 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-tmp-dir\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386730 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386700 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-model-cache\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386849 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386831 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-tmp-dir\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.386908 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.386872 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-home\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.388863 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.388838 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-dshm\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.389068 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.389049 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef2246f-256c-4851-81e9-b8dbf577ff54-tls-certs\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.445002 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.444941 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwdg\" (UniqueName: \"kubernetes.io/projected/9ef2246f-256c-4851-81e9-b8dbf577ff54-kube-api-access-dfwdg\") pod \"simulator-w1-d516c4-kserve-8d55d8669-rzps9\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.500792 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.500765 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:26.877572 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:26.877543 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9"] Oct 30 21:55:26.879089 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:55:26.879065 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ef2246f_256c_4851_81e9_b8dbf577ff54.slice/crio-03d50a2169e340b2cb0d5c22faeab108759f24be36fe7d907d416a2465ac7a93 WatchSource:0}: Error finding container 03d50a2169e340b2cb0d5c22faeab108759f24be36fe7d907d416a2465ac7a93: Status 404 returned error can't find the container with id 03d50a2169e340b2cb0d5c22faeab108759f24be36fe7d907d416a2465ac7a93 Oct 30 21:55:27.534822 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:27.534786 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" event={"ID":"9ef2246f-256c-4851-81e9-b8dbf577ff54","Type":"ContainerStarted","Data":"31c284a6a567103c5601d9d522c5d4f9040cb9fda8e4dbdbb33f1770df964d94"} Oct 30 21:55:27.534822 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:27.534827 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" event={"ID":"9ef2246f-256c-4851-81e9-b8dbf577ff54","Type":"ContainerStarted","Data":"03d50a2169e340b2cb0d5c22faeab108759f24be36fe7d907d416a2465ac7a93"} Oct 30 21:55:27.651828 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:27.651782 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podStartSLOduration=1.651767527 podStartE2EDuration="1.651767527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:27.651143515 +0000 UTC m=+776.332293411" watchObservedRunningTime="2026-10-30 21:55:27.651767527 +0000 UTC m=+776.332917420" Oct 30 21:55:28.500918 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:28.500885 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:28.549534 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:55:28.549482 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:55:31.098468 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:31.097739 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:55:36.501186 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:36.501154 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:36.513627 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:36.513599 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:36.601569 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:36.601544 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 21:55:37.999312 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:37.999281 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp"] Oct 30 21:55:38.003186 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.003169 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.096005 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.095978 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-dshm\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.096166 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.096018 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0a49b7-dea2-4923-bd0a-30e48455c795-tls-certs\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.096166 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.096052 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-home\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.096166 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.096113 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-model-cache\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.096166 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.096155 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-tmp-dir\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.096301 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.096180 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgs5\" (UniqueName: \"kubernetes.io/projected/ab0a49b7-dea2-4923-bd0a-30e48455c795-kube-api-access-zcgs5\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.109986 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.109962 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"default-dockercfg-w89h2\"" Oct 30 21:55:38.110152 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.110134 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"kube-root-ca.crt\"" Oct 30 21:55:38.111274 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.111259 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"openshift-service-ca.crt\"" Oct 30 21:55:38.111426 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.111407 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-29cbad\"/\"simulator-w6-29cbad-kserve-self-signed-certs\"" Oct 30 21:55:38.113736 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.113719 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp"] Oct 30 21:55:38.197463 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197433 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-tmp-dir\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.197644 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197476 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgs5\" (UniqueName: \"kubernetes.io/projected/ab0a49b7-dea2-4923-bd0a-30e48455c795-kube-api-access-zcgs5\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.197644 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197547 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-dshm\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.197644 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197570 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0a49b7-dea2-4923-bd0a-30e48455c795-tls-certs\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.197644 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197598 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-home\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.197644 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197623 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-model-cache\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.197924 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197909 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-tmp-dir\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.197982 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.197945 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-home\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.198071 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.198016 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-model-cache\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.199935 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.199911 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-dshm\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.200308 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.200287 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0a49b7-dea2-4923-bd0a-30e48455c795-tls-certs\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.256634 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.256567 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgs5\" (UniqueName: \"kubernetes.io/projected/ab0a49b7-dea2-4923-bd0a-30e48455c795-kube-api-access-zcgs5\") pod \"simulator-w6-29cbad-kserve-55df6bd77d-rb7fp\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.314374 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.314341 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:38.719090 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:38.719066 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp"] Oct 30 21:55:38.720363 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:55:38.720332 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab0a49b7_dea2_4923_bd0a_30e48455c795.slice/crio-d73808f22e9c9ea61e855bace0ec1b6837d5bc7503bd9eac552159b5841324bc WatchSource:0}: Error finding container d73808f22e9c9ea61e855bace0ec1b6837d5bc7503bd9eac552159b5841324bc: Status 404 returned error can't find the container with id d73808f22e9c9ea61e855bace0ec1b6837d5bc7503bd9eac552159b5841324bc Oct 30 21:55:39.608777 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:39.608735 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" event={"ID":"ab0a49b7-dea2-4923-bd0a-30e48455c795","Type":"ContainerStarted","Data":"29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d"} Oct 30 21:55:39.609187 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:39.608791 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" event={"ID":"ab0a49b7-dea2-4923-bd0a-30e48455c795","Type":"ContainerStarted","Data":"d73808f22e9c9ea61e855bace0ec1b6837d5bc7503bd9eac552159b5841324bc"} Oct 30 21:55:39.728642 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:39.728589 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podStartSLOduration=2.728575595 podStartE2EDuration="2.728575595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:39.726280157 +0000 UTC m=+788.407430050" watchObservedRunningTime="2026-10-30 21:55:39.728575595 +0000 UTC m=+788.409725488" Oct 30 21:55:40.314776 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:40.314744 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:41.121859 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.121817 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t"] Oct 30 21:55:41.125766 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.125744 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.183861 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.183833 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b0c55f\"/\"premium-w4-b0c55f-kserve-self-signed-certs\"" Oct 30 21:55:41.226302 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.226272 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrrc\" (UniqueName: \"kubernetes.io/projected/570cfea8-2743-4158-8d10-07264e9b7d9f-kube-api-access-bfrrc\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.226424 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.226313 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-model-cache\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.226424 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.226401 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-home\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.226535 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.226510 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-dshm\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.226584 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.226562 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-tmp-dir\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.226636 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.226623 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570cfea8-2743-4158-8d10-07264e9b7d9f-tls-certs\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.237011 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.236989 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t"] Oct 30 21:55:41.327259 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327221 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-dshm\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.327445 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327301 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-tmp-dir\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.327445 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327335 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570cfea8-2743-4158-8d10-07264e9b7d9f-tls-certs\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.327445 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327405 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrrc\" (UniqueName: \"kubernetes.io/projected/570cfea8-2743-4158-8d10-07264e9b7d9f-kube-api-access-bfrrc\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.327445 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327437 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-model-cache\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.327704 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327478 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-home\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.327800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327774 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-tmp-dir\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.327881 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327850 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-home\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.328010 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.327981 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-model-cache\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.329910 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.329890 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-dshm\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.330171 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.330149 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570cfea8-2743-4158-8d10-07264e9b7d9f-tls-certs\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.387531 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.387432 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrrc\" (UniqueName: \"kubernetes.io/projected/570cfea8-2743-4158-8d10-07264e9b7d9f-kube-api-access-bfrrc\") pod \"premium-w4-b0c55f-kserve-5b88cd768f-s848t\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.438352 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.438322 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:41.842689 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:41.842663 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t"] Oct 30 21:55:41.843304 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:55:41.843275 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod570cfea8_2743_4158_8d10_07264e9b7d9f.slice/crio-dfd6c776d89f6d74976b3b4c14b3940ff26046b12e63c4b9b7ed631ecce2cfbb WatchSource:0}: Error finding container dfd6c776d89f6d74976b3b4c14b3940ff26046b12e63c4b9b7ed631ecce2cfbb: Status 404 returned error can't find the container with id dfd6c776d89f6d74976b3b4c14b3940ff26046b12e63c4b9b7ed631ecce2cfbb Oct 30 21:55:42.628534 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:42.628472 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" event={"ID":"570cfea8-2743-4158-8d10-07264e9b7d9f","Type":"ContainerStarted","Data":"379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478"} Oct 30 21:55:42.628958 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:42.628543 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" event={"ID":"570cfea8-2743-4158-8d10-07264e9b7d9f","Type":"ContainerStarted","Data":"dfd6c776d89f6d74976b3b4c14b3940ff26046b12e63c4b9b7ed631ecce2cfbb"} Oct 30 21:55:42.746261 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:42.746214 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podStartSLOduration=1.7462021060000001 podStartE2EDuration="1.746202106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:42.745653459 +0000 UTC m=+791.426803353" watchObservedRunningTime="2026-10-30 21:55:42.746202106 +0000 UTC m=+791.427351998" Oct 30 21:55:43.439150 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:43.439122 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:46.088649 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.088614 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:55:46.560723 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.560692 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh"] Oct 30 21:55:46.564165 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.564143 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.621277 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.621214 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"openshift-service-ca.crt\"" Oct 30 21:55:46.668103 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.668074 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"kube-root-ca.crt\"" Oct 30 21:55:46.668278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.668198 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"simulator-w2-0c99b1-kserve-self-signed-certs\"" Oct 30 21:55:46.668278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.668259 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"default-dockercfg-cp7k7\"" Oct 30 21:55:46.674454 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.674433 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh"] Oct 30 21:55:46.684542 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.684520 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm986\" (UniqueName: \"kubernetes.io/projected/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-kube-api-access-fm986\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.684646 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.684574 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tmp-dir\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.684694 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.684672 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-home\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.684761 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.684743 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tls-certs\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.684813 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.684796 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-model-cache\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.684847 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.684824 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-dshm\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.785645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.785595 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tmp-dir\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.785827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.785682 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-home\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.785827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.785741 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tls-certs\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.785827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.785777 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-model-cache\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.785827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.785806 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-dshm\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.786046 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.785912 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm986\" (UniqueName: \"kubernetes.io/projected/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-kube-api-access-fm986\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.786109 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.786077 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tmp-dir\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.786109 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.786099 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-home\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.786213 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.786173 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-model-cache\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.788286 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.788265 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-dshm\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.788620 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.788603 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tls-certs\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.844131 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.844061 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm986\" (UniqueName: \"kubernetes.io/projected/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-kube-api-access-fm986\") pod \"simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:46.880320 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:46.880289 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:47.285104 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:47.285073 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh"] Oct 30 21:55:47.285594 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:55:47.285567 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bc0a351_9f35_447d_8f82_37a2f1d42dc4.slice/crio-55c0cf3af8967c2da9c374b182f5a1a15797ce9a40fb6c0b3d0e96d705fe6eea WatchSource:0}: Error finding container 55c0cf3af8967c2da9c374b182f5a1a15797ce9a40fb6c0b3d0e96d705fe6eea: Status 404 returned error can't find the container with id 55c0cf3af8967c2da9c374b182f5a1a15797ce9a40fb6c0b3d0e96d705fe6eea Oct 30 21:55:47.662402 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:47.662305 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" event={"ID":"7bc0a351-9f35-447d-8f82-37a2f1d42dc4","Type":"ContainerStarted","Data":"a0ba5eab8fc0aad73435234b52b2f9d252f8141ff84a49b2bb4731cbd04eff40"} Oct 30 21:55:47.662402 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:47.662359 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" event={"ID":"7bc0a351-9f35-447d-8f82-37a2f1d42dc4","Type":"ContainerStarted","Data":"55c0cf3af8967c2da9c374b182f5a1a15797ce9a40fb6c0b3d0e96d705fe6eea"} Oct 30 21:55:47.785048 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:47.784994 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podStartSLOduration=1.784980241 podStartE2EDuration="1.784980241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:47.782629281 +0000 UTC m=+796.463779185" watchObservedRunningTime="2026-10-30 21:55:47.784980241 +0000 UTC m=+796.466130134" Oct 30 21:55:47.881514 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:47.881456 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:48.314769 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:48.314735 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:48.327757 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:48.327736 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:48.679503 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:48.679416 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 21:55:51.438970 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:51.438940 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:51.451800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:51.451776 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:51.701185 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:51.701107 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 21:55:56.880438 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:56.880396 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:56.892783 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:56.892756 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:57.731107 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:55:57.731076 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 21:55:58.543100 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:55:58.543061 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:56:03.246823 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:03.246793 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:56:06.369210 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.369172 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr"] Oct 30 21:56:06.374123 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.374100 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.427401 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.427371 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0f42cb\"/\"openshift-service-ca.crt\"" Oct 30 21:56:06.427712 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.427675 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0f42cb\"/\"kube-root-ca.crt\"" Oct 30 21:56:06.478042 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.478014 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0f42cb\"/\"simulator-w3-0f42cb-kserve-self-signed-certs\"" Oct 30 21:56:06.478042 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.478020 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0f42cb\"/\"default-dockercfg-8zswv\"" Oct 30 21:56:06.480369 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.480350 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr"] Oct 30 21:56:06.482338 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.482309 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhgvt\" (UniqueName: \"kubernetes.io/projected/cc0ca645-0bf9-4e47-9cd3-222af475fff6-kube-api-access-hhgvt\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.482458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.482358 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-home\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.482458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.482374 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-dshm\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.482458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.482436 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tmp-dir\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.482608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.482592 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-model-cache\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.482681 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.482666 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tls-certs\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.583189 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.583154 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-home\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.583189 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.583199 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-dshm\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.583456 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.583217 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tmp-dir\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.583456 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.583275 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-model-cache\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.583456 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.583450 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tls-certs\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.583727 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.583701 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-home\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.583992 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.583968 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tmp-dir\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.584899 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.584869 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhgvt\" (UniqueName: \"kubernetes.io/projected/cc0ca645-0bf9-4e47-9cd3-222af475fff6-kube-api-access-hhgvt\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.585114 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.585086 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-model-cache\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.586312 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.586286 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-dshm\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.587038 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.587001 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tls-certs\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.643110 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.643039 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhgvt\" (UniqueName: \"kubernetes.io/projected/cc0ca645-0bf9-4e47-9cd3-222af475fff6-kube-api-access-hhgvt\") pod \"simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:06.685891 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:06.685855 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:07.063175 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:07.063140 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr"] Oct 30 21:56:07.064724 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:56:07.064700 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc0ca645_0bf9_4e47_9cd3_222af475fff6.slice/crio-0cb6c15c162ffbe0bedf641395f295b26ec3c261e1d1bc25ba08f21ebea9418d WatchSource:0}: Error finding container 0cb6c15c162ffbe0bedf641395f295b26ec3c261e1d1bc25ba08f21ebea9418d: Status 404 returned error can't find the container with id 0cb6c15c162ffbe0bedf641395f295b26ec3c261e1d1bc25ba08f21ebea9418d Oct 30 21:56:07.785094 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:07.785058 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" event={"ID":"cc0ca645-0bf9-4e47-9cd3-222af475fff6","Type":"ContainerStarted","Data":"b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e"} Oct 30 21:56:07.785550 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:07.785105 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" event={"ID":"cc0ca645-0bf9-4e47-9cd3-222af475fff6","Type":"ContainerStarted","Data":"0cb6c15c162ffbe0bedf641395f295b26ec3c261e1d1bc25ba08f21ebea9418d"} Oct 30 21:56:07.931341 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:07.931290 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podStartSLOduration=1.931278468 podStartE2EDuration="1.931278468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:07.929230757 +0000 UTC m=+816.610380662" watchObservedRunningTime="2026-10-30 21:56:07.931278468 +0000 UTC m=+816.612428360" Oct 30 21:56:08.686359 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:08.686324 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:16.219758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.219721 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz"] Oct 30 21:56:16.225521 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.224662 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.329064 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.329035 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"default-dockercfg-zthlq\"" Oct 30 21:56:16.329064 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.329052 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"simulator-w0-dde1e8-kserve-self-signed-certs\"" Oct 30 21:56:16.329249 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.329105 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"openshift-service-ca.crt\"" Oct 30 21:56:16.329364 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.329351 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dde1e8\"/\"kube-root-ca.crt\"" Oct 30 21:56:16.331571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.331552 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz"] Oct 30 21:56:16.382315 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.382282 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-dshm\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.382524 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.382434 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-tmp-dir\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.382524 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.382469 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/036b01a5-dc67-4162-8c8a-440786cbe489-tls-certs\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.382643 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.382533 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfkj\" (UniqueName: \"kubernetes.io/projected/036b01a5-dc67-4162-8c8a-440786cbe489-kube-api-access-fgfkj\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.382643 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.382611 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-model-cache\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.382743 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.382667 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-home\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484247 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484153 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-dshm\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484247 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484240 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-tmp-dir\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484507 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484268 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/036b01a5-dc67-4162-8c8a-440786cbe489-tls-certs\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484507 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484295 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfkj\" (UniqueName: \"kubernetes.io/projected/036b01a5-dc67-4162-8c8a-440786cbe489-kube-api-access-fgfkj\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484507 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484348 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-model-cache\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484507 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484385 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-home\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484759 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484727 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-tmp-dir\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484830 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484769 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-home\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.484947 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.484924 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-model-cache\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.486820 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.486800 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-dshm\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.487067 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.487050 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/036b01a5-dc67-4162-8c8a-440786cbe489-tls-certs\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.543228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.543194 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfkj\" (UniqueName: \"kubernetes.io/projected/036b01a5-dc67-4162-8c8a-440786cbe489-kube-api-access-fgfkj\") pod \"simulator-w0-dde1e8-kserve-69cc64f68b-fcksz\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.686137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.686102 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:16.698898 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.698874 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:16.838715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.838681 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:16.845409 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:16.845389 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 21:56:17.217679 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:56:17.217637 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod036b01a5_dc67_4162_8c8a_440786cbe489.slice/crio-cfaabb192434f4fe3182689ab00093d03c542c9467c924e7f137cb8b786aaba0 WatchSource:0}: Error finding container cfaabb192434f4fe3182689ab00093d03c542c9467c924e7f137cb8b786aaba0: Status 404 returned error can't find the container with id cfaabb192434f4fe3182689ab00093d03c542c9467c924e7f137cb8b786aaba0 Oct 30 21:56:17.217832 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:17.217791 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz"] Oct 30 21:56:17.841152 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:17.841117 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" event={"ID":"036b01a5-dc67-4162-8c8a-440786cbe489","Type":"ContainerStarted","Data":"54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d"} Oct 30 21:56:17.841656 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:17.841165 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" event={"ID":"036b01a5-dc67-4162-8c8a-440786cbe489","Type":"ContainerStarted","Data":"cfaabb192434f4fe3182689ab00093d03c542c9467c924e7f137cb8b786aaba0"} Oct 30 21:56:17.959988 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:17.959939 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podStartSLOduration=1.959922289 podStartE2EDuration="1.959922289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:17.958670955 +0000 UTC m=+826.639820829" watchObservedRunningTime="2026-10-30 21:56:17.959922289 +0000 UTC m=+826.641072186" Oct 30 21:56:18.839059 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:18.839029 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:23.971191 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:23.971157 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:56:26.839660 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:26.839629 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:26.853050 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:26.853026 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:26.909377 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:26.909354 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 21:56:28.543728 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:56:28.543670 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:56:52.849288 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:56:52.849176 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:56:58.550337 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:56:58.550300 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:57:10.766927 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.766897 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq"] Oct 30 21:57:10.770760 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.770743 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:10.825149 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.825124 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:10.875649 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.875616 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:10.875802 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.875668 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:10.875852 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.875805 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:57:10.878184 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.878162 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq"] Oct 30 21:57:10.900411 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.900390 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5kzp\" (UniqueName: \"kubernetes.io/projected/10d11304-65d3-488c-a40a-05190692e976-kube-api-access-k5kzp\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:10.900578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.900557 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d11304-65d3-488c-a40a-05190692e976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:10.900647 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:10.900634 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d11304-65d3-488c-a40a-05190692e976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.001674 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.001646 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d11304-65d3-488c-a40a-05190692e976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.001817 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.001706 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d11304-65d3-488c-a40a-05190692e976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.001817 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.001733 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5kzp\" (UniqueName: \"kubernetes.io/projected/10d11304-65d3-488c-a40a-05190692e976-kube-api-access-k5kzp\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.002322 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.002298 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d11304-65d3-488c-a40a-05190692e976-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.004193 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.004174 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d11304-65d3-488c-a40a-05190692e976-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.060861 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.060837 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5kzp\" (UniqueName: \"kubernetes.io/projected/10d11304-65d3-488c-a40a-05190692e976-kube-api-access-k5kzp\") pod \"maas-api-revoke-sub-37edb3a7-65wvq\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.083917 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.083896 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:11.470770 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:11.470748 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq"] Oct 30 21:57:11.473072 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:57:11.473046 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d11304_65d3_488c_a40a_05190692e976.slice/crio-f38983634632195d9699cf69cd34232bc623ff4f784ffbb20d80e0e985caaf77 WatchSource:0}: Error finding container f38983634632195d9699cf69cd34232bc623ff4f784ffbb20d80e0e985caaf77: Status 404 returned error can't find the container with id f38983634632195d9699cf69cd34232bc623ff4f784ffbb20d80e0e985caaf77 Oct 30 21:57:12.176655 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:12.176624 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" event={"ID":"10d11304-65d3-488c-a40a-05190692e976","Type":"ContainerStarted","Data":"f38983634632195d9699cf69cd34232bc623ff4f784ffbb20d80e0e985caaf77"} Oct 30 21:57:17.206287 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:17.206256 4933 generic.go:356] "Generic (PLEG): container finished" podID="10d11304-65d3-488c-a40a-05190692e976" containerID="dfbc4b424cad3af5d43f8ce0f20b0d3a7cef13900f5de2da77ccfe6817b31177" exitCode=0 Oct 30 21:57:17.206656 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:17.206335 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" event={"ID":"10d11304-65d3-488c-a40a-05190692e976","Type":"ContainerDied","Data":"dfbc4b424cad3af5d43f8ce0f20b0d3a7cef13900f5de2da77ccfe6817b31177"} Oct 30 21:57:18.360544 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.360513 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:18.429058 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.429030 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:57:18.469703 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.469627 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d11304-65d3-488c-a40a-05190692e976-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d11304-65d3-488c-a40a-05190692e976-maas-api-openshift-ca\") pod \"10d11304-65d3-488c-a40a-05190692e976\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " Oct 30 21:57:18.469703 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.469678 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d11304-65d3-488c-a40a-05190692e976-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d11304-65d3-488c-a40a-05190692e976-maas-api-xks-ca\") pod \"10d11304-65d3-488c-a40a-05190692e976\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " Oct 30 21:57:18.469920 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.469778 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d11304-65d3-488c-a40a-05190692e976-kube-api-access-k5kzp\" (UniqueName: \"kubernetes.io/projected/10d11304-65d3-488c-a40a-05190692e976-kube-api-access-k5kzp\") pod \"10d11304-65d3-488c-a40a-05190692e976\" (UID: \"10d11304-65d3-488c-a40a-05190692e976\") " Oct 30 21:57:18.470007 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.469984 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d11304-65d3-488c-a40a-05190692e976-maas-api-openshift-ca" pod "10d11304-65d3-488c-a40a-05190692e976" (UID: "10d11304-65d3-488c-a40a-05190692e976"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:18.470091 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.470073 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d11304-65d3-488c-a40a-05190692e976-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:18.472106 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.472074 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d11304-65d3-488c-a40a-05190692e976-maas-api-xks-ca" pod "10d11304-65d3-488c-a40a-05190692e976" (UID: "10d11304-65d3-488c-a40a-05190692e976"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:18.472106 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.472079 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d11304-65d3-488c-a40a-05190692e976-kube-api-access-k5kzp" pod "10d11304-65d3-488c-a40a-05190692e976" (UID: "10d11304-65d3-488c-a40a-05190692e976"). InnerVolumeSpecName "kube-api-access-k5kzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:18.571449 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.571389 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5kzp\" (UniqueName: \"kubernetes.io/projected/10d11304-65d3-488c-a40a-05190692e976-kube-api-access-k5kzp\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:18.571449 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:18.571422 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d11304-65d3-488c-a40a-05190692e976-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:19.220479 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:19.220452 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" Oct 30 21:57:19.220676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:19.220504 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq" event={"ID":"10d11304-65d3-488c-a40a-05190692e976","Type":"ContainerDied","Data":"f38983634632195d9699cf69cd34232bc623ff4f784ffbb20d80e0e985caaf77"} Oct 30 21:57:19.220676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:19.220553 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f38983634632195d9699cf69cd34232bc623ff4f784ffbb20d80e0e985caaf77" Oct 30 21:57:20.090013 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.089975 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4"] Oct 30 21:57:20.090874 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.090686 4933 cpu_manager.go:414] "containerMap: removing container" podUID="10d11304-65d3-488c-a40a-05190692e976" containerName="revoke-keys" Oct 30 21:57:20.090874 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.090707 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d11304-65d3-488c-a40a-05190692e976" containerName="revoke-keys" Oct 30 21:57:20.090874 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.090818 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d11304-65d3-488c-a40a-05190692e976" containerName="revoke-keys" Oct 30 21:57:20.094178 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.094162 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.172019 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.171987 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4"] Oct 30 21:57:20.188847 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.188817 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-dshm\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.188847 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.188850 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-home\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.189049 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.188944 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tls-certs\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.189049 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.189023 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-model-cache\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.189134 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.189064 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tmp-dir\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.189134 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.189099 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhfp\" (UniqueName: \"kubernetes.io/projected/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-kube-api-access-dlhfp\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.200861 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.200840 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0c99b1\"/\"premium-w2-0c99b1-kserve-self-signed-certs\"" Oct 30 21:57:20.290565 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.290521 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-dshm\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.290565 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.290556 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-home\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.290827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.290588 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tls-certs\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.290827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.290717 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-model-cache\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.290827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.290771 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tmp-dir\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.290827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.290813 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhfp\" (UniqueName: \"kubernetes.io/projected/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-kube-api-access-dlhfp\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.291049 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.291034 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-home\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.291108 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.291077 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-model-cache\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.291172 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.291150 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tmp-dir\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.293095 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.293072 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-dshm\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.293253 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.293236 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tls-certs\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.350162 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.350087 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhfp\" (UniqueName: \"kubernetes.io/projected/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-kube-api-access-dlhfp\") pod \"premium-w2-0c99b1-kserve-867d497754-vzkv4\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.406680 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.406649 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:20.828430 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:20.828407 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4"] Oct 30 21:57:20.829191 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:57:20.829166 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83ea6f0f_a024_44c5_b341_a96cd0b4aa24.slice/crio-fe43d1532a8671f74e8447969d453d80c0536a402706f1c9db47c012c67e66b6 WatchSource:0}: Error finding container fe43d1532a8671f74e8447969d453d80c0536a402706f1c9db47c012c67e66b6: Status 404 returned error can't find the container with id fe43d1532a8671f74e8447969d453d80c0536a402706f1c9db47c012c67e66b6 Oct 30 21:57:21.234139 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:21.234044 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" event={"ID":"83ea6f0f-a024-44c5-b341-a96cd0b4aa24","Type":"ContainerStarted","Data":"abf09d5425c89b81aeb1e39a4d580c4d53effc6095cbf6242773ad9399f409d2"} Oct 30 21:57:21.234139 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:21.234092 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" event={"ID":"83ea6f0f-a024-44c5-b341-a96cd0b4aa24","Type":"ContainerStarted","Data":"fe43d1532a8671f74e8447969d453d80c0536a402706f1c9db47c012c67e66b6"} Oct 30 21:57:21.357797 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:21.357726 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podStartSLOduration=1.35770548 podStartE2EDuration="1.35770548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:21.353889779 +0000 UTC m=+890.035039673" watchObservedRunningTime="2026-10-30 21:57:21.35770548 +0000 UTC m=+890.038855375" Oct 30 21:57:21.406940 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:21.406904 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:28.546508 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:57:28.545482 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:57:29.266130 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.266099 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl"] Oct 30 21:57:29.270763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.270746 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.286967 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.286946 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-dshm\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.287074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.286979 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tmp-dir\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.287074 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.287035 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-model-cache\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.287146 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.287134 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tls-certs\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.287194 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.287182 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-home\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.287245 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.287233 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs68h\" (UniqueName: \"kubernetes.io/projected/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-kube-api-access-bs68h\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.325187 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.325162 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl"] Oct 30 21:57:29.372987 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.372961 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0f42cb\"/\"premium-w3-0f42cb-kserve-self-signed-certs\"" Oct 30 21:57:29.387904 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.387875 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-model-cache\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388039 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.387909 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tls-certs\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388039 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.387941 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-home\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388039 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.388017 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs68h\" (UniqueName: \"kubernetes.io/projected/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-kube-api-access-bs68h\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388208 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.388114 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-dshm\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388208 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.388142 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tmp-dir\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.388380 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-model-cache\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.388404 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-home\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.388473 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.388469 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tmp-dir\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.390457 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.390435 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-dshm\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.390635 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.390618 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tls-certs\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.447953 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.447924 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs68h\" (UniqueName: \"kubernetes.io/projected/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-kube-api-access-bs68h\") pod \"premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.586317 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.586277 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:29.984563 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:29.984535 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl"] Oct 30 21:57:29.985371 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:57:29.985345 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d6faa2f_6fd1_43cd_8d36_d020bab4b638.slice/crio-a51c2dfcf93cb907f8d577fe8f203ca7448b0bea18745fd1dda814401b2981f5 WatchSource:0}: Error finding container a51c2dfcf93cb907f8d577fe8f203ca7448b0bea18745fd1dda814401b2981f5: Status 404 returned error can't find the container with id a51c2dfcf93cb907f8d577fe8f203ca7448b0bea18745fd1dda814401b2981f5 Oct 30 21:57:30.295070 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:30.295042 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" event={"ID":"0d6faa2f-6fd1-43cd-8d36-d020bab4b638","Type":"ContainerStarted","Data":"8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6"} Oct 30 21:57:30.295254 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:30.295081 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" event={"ID":"0d6faa2f-6fd1-43cd-8d36-d020bab4b638","Type":"ContainerStarted","Data":"a51c2dfcf93cb907f8d577fe8f203ca7448b0bea18745fd1dda814401b2981f5"} Oct 30 21:57:30.407135 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:30.407104 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:30.419695 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:30.419662 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:30.421091 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:30.421047 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podStartSLOduration=1.4210342759999999 podStartE2EDuration="1.421034276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:30.420595952 +0000 UTC m=+899.101745845" watchObservedRunningTime="2026-10-30 21:57:30.421034276 +0000 UTC m=+899.102184163" Oct 30 21:57:30.586475 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:30.586362 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:31.324525 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:31.321997 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 21:57:31.617848 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:31.617524 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:57:31.621282 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:31.621259 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 21:57:39.586719 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:39.586689 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:39.598799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:39.598776 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:40.382858 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:40.382828 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 21:57:41.327264 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.327232 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx"] Oct 30 21:57:41.331582 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.331547 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.390082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.390052 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx"] Oct 30 21:57:41.418361 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.418317 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.418558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.418418 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.418558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.418449 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgxff\" (UniqueName: \"kubernetes.io/projected/724df9e7-a661-4d50-8929-8a1f6d499bf4-kube-api-access-mgxff\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.436890 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.436861 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:41.437036 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.436911 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:41.437036 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.436867 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:57:41.437036 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.436956 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:41.519297 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.519270 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.519443 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.519425 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.519514 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.519458 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgxff\" (UniqueName: \"kubernetes.io/projected/724df9e7-a661-4d50-8929-8a1f6d499bf4-kube-api-access-mgxff\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.520082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.520049 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.521774 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.521756 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.577603 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.577539 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgxff\" (UniqueName: \"kubernetes.io/projected/724df9e7-a661-4d50-8929-8a1f6d499bf4-kube-api-access-mgxff\") pod \"maas-api-revoke-sub-9c7bc7fd-vtlsx\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.645571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.645538 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:41.708875 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:41.708236 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:57:42.026877 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:42.026833 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx"] Oct 30 21:57:42.029765 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:57:42.029256 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724df9e7_a661_4d50_8929_8a1f6d499bf4.slice/crio-1bbe46368cbcdbd08e8926c2ddb3e8d204f1cc8499fe98c2ab6582ff1eb7a766 WatchSource:0}: Error finding container 1bbe46368cbcdbd08e8926c2ddb3e8d204f1cc8499fe98c2ab6582ff1eb7a766: Status 404 returned error can't find the container with id 1bbe46368cbcdbd08e8926c2ddb3e8d204f1cc8499fe98c2ab6582ff1eb7a766 Oct 30 21:57:42.376213 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:42.376176 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" event={"ID":"724df9e7-a661-4d50-8929-8a1f6d499bf4","Type":"ContainerStarted","Data":"765d7968ebdac621414903a1a4147ec341cf0806e3143c3aea2d4dae063a54f1"} Oct 30 21:57:42.376583 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:42.376274 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" event={"ID":"724df9e7-a661-4d50-8929-8a1f6d499bf4","Type":"ContainerStarted","Data":"1bbe46368cbcdbd08e8926c2ddb3e8d204f1cc8499fe98c2ab6582ff1eb7a766"} Oct 30 21:57:42.493921 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:42.493871 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" podStartSLOduration=1.493853934 podStartE2EDuration="1.493853934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:42.491505388 +0000 UTC m=+911.172655280" watchObservedRunningTime="2026-10-30 21:57:42.493853934 +0000 UTC m=+911.175003827" Oct 30 21:57:43.385894 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:43.385806 4933 generic.go:356] "Generic (PLEG): container finished" podID="724df9e7-a661-4d50-8929-8a1f6d499bf4" containerID="765d7968ebdac621414903a1a4147ec341cf0806e3143c3aea2d4dae063a54f1" exitCode=0 Oct 30 21:57:43.385894 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:43.385886 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" event={"ID":"724df9e7-a661-4d50-8929-8a1f6d499bf4","Type":"ContainerDied","Data":"765d7968ebdac621414903a1a4147ec341cf0806e3143c3aea2d4dae063a54f1"} Oct 30 21:57:44.551299 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.551276 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:44.664779 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.664689 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724df9e7-a661-4d50-8929-8a1f6d499bf4-kube-api-access-mgxff\" (UniqueName: \"kubernetes.io/projected/724df9e7-a661-4d50-8929-8a1f6d499bf4-kube-api-access-mgxff\") pod \"724df9e7-a661-4d50-8929-8a1f6d499bf4\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " Oct 30 21:57:44.664959 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.664798 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-xks-ca\") pod \"724df9e7-a661-4d50-8929-8a1f6d499bf4\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " Oct 30 21:57:44.664959 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.664839 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-openshift-ca\") pod \"724df9e7-a661-4d50-8929-8a1f6d499bf4\" (UID: \"724df9e7-a661-4d50-8929-8a1f6d499bf4\") " Oct 30 21:57:44.665255 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.665221 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-openshift-ca" pod "724df9e7-a661-4d50-8929-8a1f6d499bf4" (UID: "724df9e7-a661-4d50-8929-8a1f6d499bf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:44.667475 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.667445 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724df9e7-a661-4d50-8929-8a1f6d499bf4-kube-api-access-mgxff" pod "724df9e7-a661-4d50-8929-8a1f6d499bf4" (UID: "724df9e7-a661-4d50-8929-8a1f6d499bf4"). InnerVolumeSpecName "kube-api-access-mgxff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:44.667754 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.667697 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-xks-ca" pod "724df9e7-a661-4d50-8929-8a1f6d499bf4" (UID: "724df9e7-a661-4d50-8929-8a1f6d499bf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:44.766391 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.766358 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgxff\" (UniqueName: \"kubernetes.io/projected/724df9e7-a661-4d50-8929-8a1f6d499bf4-kube-api-access-mgxff\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:44.766391 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.766389 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:44.766391 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:44.766399 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724df9e7-a661-4d50-8929-8a1f6d499bf4-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:45.399561 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.399532 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" Oct 30 21:57:45.399744 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.399536 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx" event={"ID":"724df9e7-a661-4d50-8929-8a1f6d499bf4","Type":"ContainerDied","Data":"1bbe46368cbcdbd08e8926c2ddb3e8d204f1cc8499fe98c2ab6582ff1eb7a766"} Oct 30 21:57:45.399744 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.399683 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bbe46368cbcdbd08e8926c2ddb3e8d204f1cc8499fe98c2ab6582ff1eb7a766" Oct 30 21:57:45.620207 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.620175 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t"] Oct 30 21:57:45.620667 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.620650 4933 cpu_manager.go:414] "containerMap: removing container" podUID="724df9e7-a661-4d50-8929-8a1f6d499bf4" containerName="revoke-keys" Oct 30 21:57:45.620711 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.620669 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724df9e7-a661-4d50-8929-8a1f6d499bf4" containerName="revoke-keys" Oct 30 21:57:45.620784 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.620775 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="724df9e7-a661-4d50-8929-8a1f6d499bf4" containerName="revoke-keys" Oct 30 21:57:45.623902 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.623880 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.678246 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.678166 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:45.678391 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.678257 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:45.678468 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.678184 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:45.679298 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.679257 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfgq5\" (UniqueName: \"kubernetes.io/projected/06005574-ed44-4fec-83bd-7964ebb58b66-kube-api-access-rfgq5\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.679421 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.679348 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.679548 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.679437 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.681976 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.681949 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t"] Oct 30 21:57:45.728392 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.728361 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:57:45.780558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.780527 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.780716 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.780590 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.780716 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.780648 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfgq5\" (UniqueName: \"kubernetes.io/projected/06005574-ed44-4fec-83bd-7964ebb58b66-kube-api-access-rfgq5\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.781100 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.781081 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.783100 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.783077 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.857258 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.857225 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfgq5\" (UniqueName: \"kubernetes.io/projected/06005574-ed44-4fec-83bd-7964ebb58b66-kube-api-access-rfgq5\") pod \"maas-api-revoke-sub-04e8d550-tdz7t\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:45.936975 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:45.936912 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:46.314997 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:57:46.314957 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06005574_ed44_4fec_83bd_7964ebb58b66.slice/crio-798e1809cc42803e598074108e985d5d5fa6aa4fc86264478d1810628048f386 WatchSource:0}: Error finding container 798e1809cc42803e598074108e985d5d5fa6aa4fc86264478d1810628048f386: Status 404 returned error can't find the container with id 798e1809cc42803e598074108e985d5d5fa6aa4fc86264478d1810628048f386 Oct 30 21:57:46.315155 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:46.315096 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t"] Oct 30 21:57:46.406618 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:46.406584 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" event={"ID":"06005574-ed44-4fec-83bd-7964ebb58b66","Type":"ContainerStarted","Data":"798e1809cc42803e598074108e985d5d5fa6aa4fc86264478d1810628048f386"} Oct 30 21:57:47.415645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:47.415618 4933 generic.go:356] "Generic (PLEG): container finished" podID="06005574-ed44-4fec-83bd-7964ebb58b66" containerID="bf94123bd281eccf089930fcfe2d6a6de8fc720ce21f3bf4b88e2bbdb27b1e19" exitCode=0 Oct 30 21:57:47.416034 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:47.415711 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" event={"ID":"06005574-ed44-4fec-83bd-7964ebb58b66","Type":"ContainerDied","Data":"bf94123bd281eccf089930fcfe2d6a6de8fc720ce21f3bf4b88e2bbdb27b1e19"} Oct 30 21:57:48.573849 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.573826 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:48.708571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.708461 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-xks-ca\") pod \"06005574-ed44-4fec-83bd-7964ebb58b66\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " Oct 30 21:57:48.708571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.708565 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-openshift-ca\") pod \"06005574-ed44-4fec-83bd-7964ebb58b66\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " Oct 30 21:57:48.708805 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.708632 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06005574-ed44-4fec-83bd-7964ebb58b66-kube-api-access-rfgq5\" (UniqueName: \"kubernetes.io/projected/06005574-ed44-4fec-83bd-7964ebb58b66-kube-api-access-rfgq5\") pod \"06005574-ed44-4fec-83bd-7964ebb58b66\" (UID: \"06005574-ed44-4fec-83bd-7964ebb58b66\") " Oct 30 21:57:48.708954 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.708931 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-openshift-ca" pod "06005574-ed44-4fec-83bd-7964ebb58b66" (UID: "06005574-ed44-4fec-83bd-7964ebb58b66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:48.709052 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.709036 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:48.710896 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.710866 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-xks-ca" pod "06005574-ed44-4fec-83bd-7964ebb58b66" (UID: "06005574-ed44-4fec-83bd-7964ebb58b66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:48.711076 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.711048 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06005574-ed44-4fec-83bd-7964ebb58b66-kube-api-access-rfgq5" pod "06005574-ed44-4fec-83bd-7964ebb58b66" (UID: "06005574-ed44-4fec-83bd-7964ebb58b66"). InnerVolumeSpecName "kube-api-access-rfgq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:48.809612 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.809582 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfgq5\" (UniqueName: \"kubernetes.io/projected/06005574-ed44-4fec-83bd-7964ebb58b66-kube-api-access-rfgq5\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:48.809612 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:48.809608 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06005574-ed44-4fec-83bd-7964ebb58b66-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:49.429828 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:49.429807 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" Oct 30 21:57:49.430039 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:49.429806 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t" event={"ID":"06005574-ed44-4fec-83bd-7964ebb58b66","Type":"ContainerDied","Data":"798e1809cc42803e598074108e985d5d5fa6aa4fc86264478d1810628048f386"} Oct 30 21:57:49.430039 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:49.429965 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798e1809cc42803e598074108e985d5d5fa6aa4fc86264478d1810628048f386" Oct 30 21:57:56.730961 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.730928 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs"] Oct 30 21:57:56.731508 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.731466 4933 cpu_manager.go:414] "containerMap: removing container" podUID="06005574-ed44-4fec-83bd-7964ebb58b66" containerName="revoke-keys" Oct 30 21:57:56.731620 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.731515 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06005574-ed44-4fec-83bd-7964ebb58b66" containerName="revoke-keys" Oct 30 21:57:56.731663 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.731641 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="06005574-ed44-4fec-83bd-7964ebb58b66" containerName="revoke-keys" Oct 30 21:57:56.735137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.735122 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.791721 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.791684 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs"] Oct 30 21:57:56.838623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.838594 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:56.838623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.838606 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:56.838623 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.838625 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:56.838829 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.838666 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:57:56.885597 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.885566 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.885764 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.885740 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhf48\" (UniqueName: \"kubernetes.io/projected/428266de-5106-4b55-bbe1-64e9ee734f75-kube-api-access-fhf48\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.885834 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.885799 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.987044 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.986966 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhf48\" (UniqueName: \"kubernetes.io/projected/428266de-5106-4b55-bbe1-64e9ee734f75-kube-api-access-fhf48\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.987044 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.987005 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.987253 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.987070 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.987823 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.987802 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:56.989682 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:56.989661 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:57.045388 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:57.045357 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhf48\" (UniqueName: \"kubernetes.io/projected/428266de-5106-4b55-bbe1-64e9ee734f75-kube-api-access-fhf48\") pod \"maas-api-revoke-sub-e490d292-4blrs\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:57.051027 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:57.051004 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:57.433561 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:57:57.433535 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428266de_5106_4b55_bbe1_64e9ee734f75.slice/crio-df74b0d12d2bc2c3ae86148fda4c4ed7cb4f11d44f4ab6b0d9b29bd8a6c236fa WatchSource:0}: Error finding container df74b0d12d2bc2c3ae86148fda4c4ed7cb4f11d44f4ab6b0d9b29bd8a6c236fa: Status 404 returned error can't find the container with id df74b0d12d2bc2c3ae86148fda4c4ed7cb4f11d44f4ab6b0d9b29bd8a6c236fa Oct 30 21:57:57.434020 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:57.433993 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs"] Oct 30 21:57:57.480073 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:57.480048 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" event={"ID":"428266de-5106-4b55-bbe1-64e9ee734f75","Type":"ContainerStarted","Data":"df74b0d12d2bc2c3ae86148fda4c4ed7cb4f11d44f4ab6b0d9b29bd8a6c236fa"} Oct 30 21:57:58.487909 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:58.487881 4933 generic.go:356] "Generic (PLEG): container finished" podID="428266de-5106-4b55-bbe1-64e9ee734f75" containerID="a655fd2fccf85076bd47835c6632d953759f675121f40efed00520bb330992bc" exitCode=0 Oct 30 21:57:58.488337 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:58.487963 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" event={"ID":"428266de-5106-4b55-bbe1-64e9ee734f75","Type":"ContainerDied","Data":"a655fd2fccf85076bd47835c6632d953759f675121f40efed00520bb330992bc"} Oct 30 21:57:58.543965 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:57:58.543931 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:57:59.643604 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.643577 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:57:59.814829 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.814797 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428266de-5106-4b55-bbe1-64e9ee734f75-kube-api-access-fhf48\" (UniqueName: \"kubernetes.io/projected/428266de-5106-4b55-bbe1-64e9ee734f75-kube-api-access-fhf48\") pod \"428266de-5106-4b55-bbe1-64e9ee734f75\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " Oct 30 21:57:59.814963 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.814863 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-openshift-ca\") pod \"428266de-5106-4b55-bbe1-64e9ee734f75\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " Oct 30 21:57:59.815013 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.814997 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-xks-ca\") pod \"428266de-5106-4b55-bbe1-64e9ee734f75\" (UID: \"428266de-5106-4b55-bbe1-64e9ee734f75\") " Oct 30 21:57:59.815195 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.815175 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-openshift-ca" pod "428266de-5106-4b55-bbe1-64e9ee734f75" (UID: "428266de-5106-4b55-bbe1-64e9ee734f75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:59.815321 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.815308 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:59.817097 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.817071 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-xks-ca" pod "428266de-5106-4b55-bbe1-64e9ee734f75" (UID: "428266de-5106-4b55-bbe1-64e9ee734f75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:59.817191 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.817111 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428266de-5106-4b55-bbe1-64e9ee734f75-kube-api-access-fhf48" pod "428266de-5106-4b55-bbe1-64e9ee734f75" (UID: "428266de-5106-4b55-bbe1-64e9ee734f75"). InnerVolumeSpecName "kube-api-access-fhf48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:59.916192 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.916169 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhf48\" (UniqueName: \"kubernetes.io/projected/428266de-5106-4b55-bbe1-64e9ee734f75-kube-api-access-fhf48\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:57:59.916192 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:57:59.916194 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/428266de-5106-4b55-bbe1-64e9ee734f75-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:00.502182 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:00.502144 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" event={"ID":"428266de-5106-4b55-bbe1-64e9ee734f75","Type":"ContainerDied","Data":"df74b0d12d2bc2c3ae86148fda4c4ed7cb4f11d44f4ab6b0d9b29bd8a6c236fa"} Oct 30 21:58:00.502326 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:00.502204 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df74b0d12d2bc2c3ae86148fda4c4ed7cb4f11d44f4ab6b0d9b29bd8a6c236fa" Oct 30 21:58:00.502326 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:00.502161 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs" Oct 30 21:58:01.240553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.240518 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6"] Oct 30 21:58:01.241140 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.241120 4933 cpu_manager.go:414] "containerMap: removing container" podUID="428266de-5106-4b55-bbe1-64e9ee734f75" containerName="revoke-keys" Oct 30 21:58:01.241224 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.241142 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428266de-5106-4b55-bbe1-64e9ee734f75" containerName="revoke-keys" Oct 30 21:58:01.241294 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.241278 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="428266de-5106-4b55-bbe1-64e9ee734f75" containerName="revoke-keys" Oct 30 21:58:01.244617 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.244595 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.300436 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.300408 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6"] Oct 30 21:58:01.329423 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.329395 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.329610 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.329444 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.329610 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.329567 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkw2\" (UniqueName: \"kubernetes.io/projected/e7a8239a-4975-4b5b-92aa-7b00144c9bad-kube-api-access-mtkw2\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.347974 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.347946 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:01.348141 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.347949 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:01.348141 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.347972 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:01.348293 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.348164 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:58:01.430191 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.430158 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.430370 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.430224 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.430370 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.430267 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkw2\" (UniqueName: \"kubernetes.io/projected/e7a8239a-4975-4b5b-92aa-7b00144c9bad-kube-api-access-mtkw2\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.431033 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.430997 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.432872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.432843 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.488464 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.488432 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkw2\" (UniqueName: \"kubernetes.io/projected/e7a8239a-4975-4b5b-92aa-7b00144c9bad-kube-api-access-mtkw2\") pod \"maas-api-revoke-sub-4e44eff9-d7cw6\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.561406 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.561369 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:01.938976 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:01.938940 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6"] Oct 30 21:58:01.943746 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:01.943502 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a8239a_4975_4b5b_92aa_7b00144c9bad.slice/crio-119496a9a63e65c740ecb86ce098e507218fd78809825d8effcac2f76a5a3d55 WatchSource:0}: Error finding container 119496a9a63e65c740ecb86ce098e507218fd78809825d8effcac2f76a5a3d55: Status 404 returned error can't find the container with id 119496a9a63e65c740ecb86ce098e507218fd78809825d8effcac2f76a5a3d55 Oct 30 21:58:02.515629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:02.515550 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" event={"ID":"e7a8239a-4975-4b5b-92aa-7b00144c9bad","Type":"ContainerStarted","Data":"1f0cb1d762b457ebc1c8f3150f204f18a3cac2ddd36318d8b7e44a820eff1bfe"} Oct 30 21:58:02.515629 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:02.515617 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" event={"ID":"e7a8239a-4975-4b5b-92aa-7b00144c9bad","Type":"ContainerStarted","Data":"119496a9a63e65c740ecb86ce098e507218fd78809825d8effcac2f76a5a3d55"} Oct 30 21:58:02.634210 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:02.634157 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" podStartSLOduration=1.634143299 podStartE2EDuration="1.634143299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:02.633054813 +0000 UTC m=+931.314204703" watchObservedRunningTime="2026-10-30 21:58:02.634143299 +0000 UTC m=+931.315293244" Oct 30 21:58:03.522980 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:03.522948 4933 generic.go:356] "Generic (PLEG): container finished" podID="e7a8239a-4975-4b5b-92aa-7b00144c9bad" containerID="1f0cb1d762b457ebc1c8f3150f204f18a3cac2ddd36318d8b7e44a820eff1bfe" exitCode=0 Oct 30 21:58:03.523359 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:03.523029 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" event={"ID":"e7a8239a-4975-4b5b-92aa-7b00144c9bad","Type":"ContainerDied","Data":"1f0cb1d762b457ebc1c8f3150f204f18a3cac2ddd36318d8b7e44a820eff1bfe"} Oct 30 21:58:04.679405 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.679374 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:04.870884 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.870815 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-openshift-ca\") pod \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " Oct 30 21:58:04.871027 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.870933 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-xks-ca\") pod \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " Oct 30 21:58:04.871072 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.871041 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a8239a-4975-4b5b-92aa-7b00144c9bad-kube-api-access-mtkw2\" (UniqueName: \"kubernetes.io/projected/e7a8239a-4975-4b5b-92aa-7b00144c9bad-kube-api-access-mtkw2\") pod \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\" (UID: \"e7a8239a-4975-4b5b-92aa-7b00144c9bad\") " Oct 30 21:58:04.871216 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.871194 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-openshift-ca" pod "e7a8239a-4975-4b5b-92aa-7b00144c9bad" (UID: "e7a8239a-4975-4b5b-92aa-7b00144c9bad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:04.871377 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.871362 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.873090 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.873063 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-xks-ca" pod "e7a8239a-4975-4b5b-92aa-7b00144c9bad" (UID: "e7a8239a-4975-4b5b-92aa-7b00144c9bad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:04.873178 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.873153 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a8239a-4975-4b5b-92aa-7b00144c9bad-kube-api-access-mtkw2" pod "e7a8239a-4975-4b5b-92aa-7b00144c9bad" (UID: "e7a8239a-4975-4b5b-92aa-7b00144c9bad"). InnerVolumeSpecName "kube-api-access-mtkw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:04.972092 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.972063 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkw2\" (UniqueName: \"kubernetes.io/projected/e7a8239a-4975-4b5b-92aa-7b00144c9bad-kube-api-access-mtkw2\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.972092 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:04.972087 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7a8239a-4975-4b5b-92aa-7b00144c9bad-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:05.534893 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:05.534869 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" Oct 30 21:58:05.534893 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:05.534873 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6" event={"ID":"e7a8239a-4975-4b5b-92aa-7b00144c9bad","Type":"ContainerDied","Data":"119496a9a63e65c740ecb86ce098e507218fd78809825d8effcac2f76a5a3d55"} Oct 30 21:58:05.535062 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:05.534920 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119496a9a63e65c740ecb86ce098e507218fd78809825d8effcac2f76a5a3d55" Oct 30 21:58:16.953619 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:16.953545 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp"] Oct 30 21:58:16.954194 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:16.954180 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e7a8239a-4975-4b5b-92aa-7b00144c9bad" containerName="revoke-keys" Oct 30 21:58:16.954240 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:16.954197 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a8239a-4975-4b5b-92aa-7b00144c9bad" containerName="revoke-keys" Oct 30 21:58:16.954355 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:16.954345 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a8239a-4975-4b5b-92aa-7b00144c9bad" containerName="revoke-keys" Oct 30 21:58:16.957844 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:16.957826 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:16.999949 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:16.999924 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.000063 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:16.999981 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.000115 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.000093 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvj7\" (UniqueName: \"kubernetes.io/projected/aeb4666e-865e-4c47-8b78-6b13f7601b8d-kube-api-access-cwvj7\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.013557 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.013532 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp"] Oct 30 21:58:17.060872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.060843 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:17.061018 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.060975 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:17.061072 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.061031 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:17.061749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.061728 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:58:17.101393 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.101369 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.101584 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.101413 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvj7\" (UniqueName: \"kubernetes.io/projected/aeb4666e-865e-4c47-8b78-6b13f7601b8d-kube-api-access-cwvj7\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.101584 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.101477 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.102220 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.102149 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.104075 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.104046 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.160535 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.160510 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvj7\" (UniqueName: \"kubernetes.io/projected/aeb4666e-865e-4c47-8b78-6b13f7601b8d-kube-api-access-cwvj7\") pod \"maas-api-revoke-sub-1013e658-dsmxp\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.272981 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.272910 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:17.661239 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.661206 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp"] Oct 30 21:58:17.662034 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:17.661997 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb4666e_865e_4c47_8b78_6b13f7601b8d.slice/crio-df39edbd71103af504cf5278ff33a858d96a5a2719dda72e85447996dd15a9f3 WatchSource:0}: Error finding container df39edbd71103af504cf5278ff33a858d96a5a2719dda72e85447996dd15a9f3: Status 404 returned error can't find the container with id df39edbd71103af504cf5278ff33a858d96a5a2719dda72e85447996dd15a9f3 Oct 30 21:58:17.812006 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:17.811938 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:58:18.626468 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.626434 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" event={"ID":"aeb4666e-865e-4c47-8b78-6b13f7601b8d","Type":"ContainerStarted","Data":"ea8a07a584f5e9a44365b7e7937b58533da6be4e2090adcc45ed0afa2753b6db"} Oct 30 21:58:18.627029 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.626997 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" event={"ID":"aeb4666e-865e-4c47-8b78-6b13f7601b8d","Type":"ContainerStarted","Data":"df39edbd71103af504cf5278ff33a858d96a5a2719dda72e85447996dd15a9f3"} Oct 30 21:58:18.736403 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.736365 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm"] Oct 30 21:58:18.740102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.740082 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.743980 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.743932 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" podStartSLOduration=2.743915394 podStartE2EDuration="2.743915394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:18.743233809 +0000 UTC m=+947.424383715" watchObservedRunningTime="2026-10-30 21:58:18.743915394 +0000 UTC m=+947.425065288" Oct 30 21:58:18.797675 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.797642 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm"] Oct 30 21:58:18.825593 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.825548 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.825765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.825747 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.825818 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.825799 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wr2d\" (UniqueName: \"kubernetes.io/projected/55152a9c-b3db-4e2a-a309-c493e0d84ad8-kube-api-access-9wr2d\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.927215 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.927129 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.927388 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.927278 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.927388 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.927322 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wr2d\" (UniqueName: \"kubernetes.io/projected/55152a9c-b3db-4e2a-a309-c493e0d84ad8-kube-api-access-9wr2d\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.928040 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.927957 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:18.929866 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:18.929842 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:19.000741 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:19.000710 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wr2d\" (UniqueName: \"kubernetes.io/projected/55152a9c-b3db-4e2a-a309-c493e0d84ad8-kube-api-access-9wr2d\") pod \"maas-api-revoke-sub-548685a1-s5thm\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:19.115036 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:19.115008 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:19.535749 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:19.535711 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55152a9c_b3db_4e2a_a309_c493e0d84ad8.slice/crio-3e7726e9c183634ddf5dadbfcf8f60929d104506376b2c18893034a4054b091b WatchSource:0}: Error finding container 3e7726e9c183634ddf5dadbfcf8f60929d104506376b2c18893034a4054b091b: Status 404 returned error can't find the container with id 3e7726e9c183634ddf5dadbfcf8f60929d104506376b2c18893034a4054b091b Oct 30 21:58:19.544086 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:19.541967 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm"] Oct 30 21:58:19.638300 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:19.638255 4933 generic.go:356] "Generic (PLEG): container finished" podID="aeb4666e-865e-4c47-8b78-6b13f7601b8d" containerID="ea8a07a584f5e9a44365b7e7937b58533da6be4e2090adcc45ed0afa2753b6db" exitCode=0 Oct 30 21:58:19.638711 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:19.638384 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" event={"ID":"aeb4666e-865e-4c47-8b78-6b13f7601b8d","Type":"ContainerDied","Data":"ea8a07a584f5e9a44365b7e7937b58533da6be4e2090adcc45ed0afa2753b6db"} Oct 30 21:58:19.640019 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:19.639990 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" event={"ID":"55152a9c-b3db-4e2a-a309-c493e0d84ad8","Type":"ContainerStarted","Data":"3e7726e9c183634ddf5dadbfcf8f60929d104506376b2c18893034a4054b091b"} Oct 30 21:58:20.649240 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.649199 4933 generic.go:356] "Generic (PLEG): container finished" podID="55152a9c-b3db-4e2a-a309-c493e0d84ad8" containerID="673ae5df906a2c9e163deba50e6a1ec8581716f1447c2b0e60519de0967497fa" exitCode=0 Oct 30 21:58:20.649644 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.649265 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" event={"ID":"55152a9c-b3db-4e2a-a309-c493e0d84ad8","Type":"ContainerDied","Data":"673ae5df906a2c9e163deba50e6a1ec8581716f1447c2b0e60519de0967497fa"} Oct 30 21:58:20.820388 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.820346 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:20.857849 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.857818 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-openshift-ca\") pod \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " Oct 30 21:58:20.858014 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.857870 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-xks-ca\") pod \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " Oct 30 21:58:20.858014 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.857962 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb4666e-865e-4c47-8b78-6b13f7601b8d-kube-api-access-cwvj7\" (UniqueName: \"kubernetes.io/projected/aeb4666e-865e-4c47-8b78-6b13f7601b8d-kube-api-access-cwvj7\") pod \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\" (UID: \"aeb4666e-865e-4c47-8b78-6b13f7601b8d\") " Oct 30 21:58:20.858192 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.858156 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-openshift-ca" pod "aeb4666e-865e-4c47-8b78-6b13f7601b8d" (UID: "aeb4666e-865e-4c47-8b78-6b13f7601b8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:20.860136 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.860106 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-xks-ca" pod "aeb4666e-865e-4c47-8b78-6b13f7601b8d" (UID: "aeb4666e-865e-4c47-8b78-6b13f7601b8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:20.860136 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.860126 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb4666e-865e-4c47-8b78-6b13f7601b8d-kube-api-access-cwvj7" pod "aeb4666e-865e-4c47-8b78-6b13f7601b8d" (UID: "aeb4666e-865e-4c47-8b78-6b13f7601b8d"). InnerVolumeSpecName "kube-api-access-cwvj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:20.959246 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.959209 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:20.959246 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.959248 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aeb4666e-865e-4c47-8b78-6b13f7601b8d-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:20.959525 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:20.959268 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwvj7\" (UniqueName: \"kubernetes.io/projected/aeb4666e-865e-4c47-8b78-6b13f7601b8d-kube-api-access-cwvj7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:21.656761 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.656729 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" Oct 30 21:58:21.657157 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.656763 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp" event={"ID":"aeb4666e-865e-4c47-8b78-6b13f7601b8d","Type":"ContainerDied","Data":"df39edbd71103af504cf5278ff33a858d96a5a2719dda72e85447996dd15a9f3"} Oct 30 21:58:21.657157 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.656818 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df39edbd71103af504cf5278ff33a858d96a5a2719dda72e85447996dd15a9f3" Oct 30 21:58:21.829717 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.829624 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:21.867868 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.867837 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-xks-ca\") pod \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " Oct 30 21:58:21.868050 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.867980 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55152a9c-b3db-4e2a-a309-c493e0d84ad8-kube-api-access-9wr2d\" (UniqueName: \"kubernetes.io/projected/55152a9c-b3db-4e2a-a309-c493e0d84ad8-kube-api-access-9wr2d\") pod \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " Oct 30 21:58:21.868050 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.868025 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-openshift-ca\") pod \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\" (UID: \"55152a9c-b3db-4e2a-a309-c493e0d84ad8\") " Oct 30 21:58:21.868569 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.868476 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-openshift-ca" pod "55152a9c-b3db-4e2a-a309-c493e0d84ad8" (UID: "55152a9c-b3db-4e2a-a309-c493e0d84ad8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:21.870102 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.870075 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-xks-ca" pod "55152a9c-b3db-4e2a-a309-c493e0d84ad8" (UID: "55152a9c-b3db-4e2a-a309-c493e0d84ad8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:21.870198 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.870108 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55152a9c-b3db-4e2a-a309-c493e0d84ad8-kube-api-access-9wr2d" pod "55152a9c-b3db-4e2a-a309-c493e0d84ad8" (UID: "55152a9c-b3db-4e2a-a309-c493e0d84ad8"). InnerVolumeSpecName "kube-api-access-9wr2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:21.969734 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.969698 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wr2d\" (UniqueName: \"kubernetes.io/projected/55152a9c-b3db-4e2a-a309-c493e0d84ad8-kube-api-access-9wr2d\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:21.969734 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.969726 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:21.969734 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:21.969737 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55152a9c-b3db-4e2a-a309-c493e0d84ad8-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:22.048071 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.047980 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44"] Oct 30 21:58:22.048512 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.048480 4933 cpu_manager.go:414] "containerMap: removing container" podUID="aeb4666e-865e-4c47-8b78-6b13f7601b8d" containerName="revoke-keys" Oct 30 21:58:22.048601 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.048512 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb4666e-865e-4c47-8b78-6b13f7601b8d" containerName="revoke-keys" Oct 30 21:58:22.048601 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.048522 4933 cpu_manager.go:414] "containerMap: removing container" podUID="55152a9c-b3db-4e2a-a309-c493e0d84ad8" containerName="revoke-keys" Oct 30 21:58:22.048601 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.048527 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55152a9c-b3db-4e2a-a309-c493e0d84ad8" containerName="revoke-keys" Oct 30 21:58:22.048761 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.048602 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb4666e-865e-4c47-8b78-6b13f7601b8d" containerName="revoke-keys" Oct 30 21:58:22.048761 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.048611 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="55152a9c-b3db-4e2a-a309-c493e0d84ad8" containerName="revoke-keys" Oct 30 21:58:22.051940 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.051921 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.108166 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.108139 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b0c55f-openshift-default-dockercfg-gqvhz\"" Oct 30 21:58:22.121897 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.121868 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44"] Oct 30 21:58:22.172025 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172001 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172173 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172056 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfsk\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-kube-api-access-lbfsk\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172173 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172087 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172173 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172147 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172317 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172180 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172317 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172267 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172335 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172407 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172388 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.172529 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.172415 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.261517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.261474 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9"] Oct 30 21:58:22.265231 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.265216 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.273877 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.273853 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274034 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.273895 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfsk\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-kube-api-access-lbfsk\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274034 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.273914 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274034 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.273940 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274034 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.273972 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274034 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.274018 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274290 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.274055 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274290 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.274097 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274290 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.274163 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.274733 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.274945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.274923 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.275024 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.274980 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.275412 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.275391 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.275640 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.275613 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.276914 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.276896 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.277169 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.277150 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.322900 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.322822 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9f8aea-openshift-default-dockercfg-64lqz\"" Oct 30 21:58:22.327100 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.327076 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9"] Oct 30 21:58:22.332585 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.332555 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.333306 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.333283 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfsk\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-kube-api-access-lbfsk\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.365599 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.365571 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:22.374709 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.374687 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e090da55-55cc-45c6-bf58-8c5f17818d13-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.374826 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.374740 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.374900 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.374856 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.374958 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.374916 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.374998 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.374966 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.374998 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.374993 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.375087 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.375008 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.375087 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.375023 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.375162 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.375138 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sw6d\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-kube-api-access-6sw6d\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476223 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476192 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476236 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476267 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476310 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476335 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476368 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476361 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476680 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476403 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sw6d\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-kube-api-access-6sw6d\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476680 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476470 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e090da55-55cc-45c6-bf58-8c5f17818d13-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476680 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476594 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476843 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476728 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.476843 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476772 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.477008 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.476987 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.477217 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.477186 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.477804 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.477779 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e090da55-55cc-45c6-bf58-8c5f17818d13-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.478754 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.478735 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.479420 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.479399 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.535393 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.535357 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.535791 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.535768 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sw6d\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-kube-api-access-6sw6d\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.582443 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.582379 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:22.665733 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.665702 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" event={"ID":"55152a9c-b3db-4e2a-a309-c493e0d84ad8","Type":"ContainerDied","Data":"3e7726e9c183634ddf5dadbfcf8f60929d104506376b2c18893034a4054b091b"} Oct 30 21:58:22.666086 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.665751 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e7726e9c183634ddf5dadbfcf8f60929d104506376b2c18893034a4054b091b" Oct 30 21:58:22.666086 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.665805 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm" Oct 30 21:58:22.809307 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:22.809279 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode067a905_6cfc_41fc_bdea_052a9f35fa8c.slice/crio-2989bdef8dc2d532d7340dc62ef2c2c9fed7bf39f967b09b11bdb7baf47d3114 WatchSource:0}: Error finding container 2989bdef8dc2d532d7340dc62ef2c2c9fed7bf39f967b09b11bdb7baf47d3114: Status 404 returned error can't find the container with id 2989bdef8dc2d532d7340dc62ef2c2c9fed7bf39f967b09b11bdb7baf47d3114 Oct 30 21:58:22.812481 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.812459 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44"] Oct 30 21:58:22.812609 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.812526 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:22.812609 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.812598 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:22.812688 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.812629 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:22.963933 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.963175 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9"] Oct 30 21:58:22.970429 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.970392 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:22.970540 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.970515 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:22.970578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:22.970559 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:23.688305 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:23.688261 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" event={"ID":"e090da55-55cc-45c6-bf58-8c5f17818d13","Type":"ContainerStarted","Data":"7ad304d198365df8584cdc82fda141af4d0e4a992a6f480a17e626ce7bca1c36"} Oct 30 21:58:23.688838 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:23.688331 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" event={"ID":"e090da55-55cc-45c6-bf58-8c5f17818d13","Type":"ContainerStarted","Data":"39353c61dacb3b0378ed81401db7f36da1caf8ed5bffb5f5aed92a96c70ac25d"} Oct 30 21:58:23.696553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:23.693843 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" event={"ID":"e067a905-6cfc-41fc-bdea-052a9f35fa8c","Type":"ContainerStarted","Data":"74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68"} Oct 30 21:58:23.696553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:23.694004 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" event={"ID":"e067a905-6cfc-41fc-bdea-052a9f35fa8c","Type":"ContainerStarted","Data":"2989bdef8dc2d532d7340dc62ef2c2c9fed7bf39f967b09b11bdb7baf47d3114"} Oct 30 21:58:23.819180 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:23.819118 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" podStartSLOduration=1.819104608 podStartE2EDuration="1.819104608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.817423975 +0000 UTC m=+952.498573869" watchObservedRunningTime="2026-10-30 21:58:23.819104608 +0000 UTC m=+952.500254501" Oct 30 21:58:23.946573 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:23.945969 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podStartSLOduration=2.94595006 podStartE2EDuration="2.94595006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:23.945816402 +0000 UTC m=+952.626966298" watchObservedRunningTime="2026-10-30 21:58:23.94595006 +0000 UTC m=+952.627099954" Oct 30 21:58:24.370518 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:24.369552 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:24.388513 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:24.380601 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:58:24.388513 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:24.380661 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:58:24.585517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:24.584468 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:58:24.585517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:24.584560 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:58:24.585517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:24.584356 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:25.371515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:25.370527 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:58:25.371515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:25.370586 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:58:25.591278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:25.584804 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:58:25.591278 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:25.584864 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:58:26.371526 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:26.370819 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:58:26.371526 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:26.370877 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:58:26.592863 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:26.592829 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:58:26.593124 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:26.593093 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:58:27.367516 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.366815 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:58:27.367516 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.366874 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:58:27.583516 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.582878 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:58:27.583516 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.582941 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:58:27.648785 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.648696 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2"] Oct 30 21:58:27.670827 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.670803 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.707877 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.707708 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2"] Oct 30 21:58:27.768515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.756811 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:27.768515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.756907 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:58:27.768515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.756935 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:27.768515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.757000 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:27.804517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.803056 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.804517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.803155 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljwg\" (UniqueName: \"kubernetes.io/projected/d01f96a5-59c1-4e35-8c31-d4a01e190821-kube-api-access-8ljwg\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.804517 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.803274 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.903919 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.903830 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljwg\" (UniqueName: \"kubernetes.io/projected/d01f96a5-59c1-4e35-8c31-d4a01e190821-kube-api-access-8ljwg\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.904082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.904013 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.904173 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.904092 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.904838 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.904810 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.906763 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.906739 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:27.983549 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:27.975107 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljwg\" (UniqueName: \"kubernetes.io/projected/d01f96a5-59c1-4e35-8c31-d4a01e190821-kube-api-access-8ljwg\") pod \"maas-api-revoke-sub-2304ee4f-grhx2\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:28.003715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:28.003685 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:28.380618 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:28.380585 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:58:28.380874 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:28.380845 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:58:28.487851 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:28.487822 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2"] Oct 30 21:58:28.492956 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:28.492930 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01f96a5_59c1_4e35_8c31_d4a01e190821.slice/crio-85522cd2f2bbbf30183c1af3ffe0f09f0f3accbef0df31507135955aa8e51b9e WatchSource:0}: Error finding container 85522cd2f2bbbf30183c1af3ffe0f09f0f3accbef0df31507135955aa8e51b9e: Status 404 returned error can't find the container with id 85522cd2f2bbbf30183c1af3ffe0f09f0f3accbef0df31507135955aa8e51b9e Oct 30 21:58:28.586311 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:28.585878 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:58:28.586311 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:28.585937 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 30 21:58:28.601552 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:58:28.601515 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:58:28.800941 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:28.800903 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" event={"ID":"d01f96a5-59c1-4e35-8c31-d4a01e190821","Type":"ContainerStarted","Data":"85522cd2f2bbbf30183c1af3ffe0f09f0f3accbef0df31507135955aa8e51b9e"} Oct 30 21:58:29.366514 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:29.366456 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:58:29.366693 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:29.366548 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:58:29.779222 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:29.779190 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:29.779967 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:29.779940 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:29.780647 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:29.780630 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 21:58:29.808946 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:29.808914 4933 generic.go:356] "Generic (PLEG): container finished" podID="d01f96a5-59c1-4e35-8c31-d4a01e190821" containerID="189cfecbf89b0d00670af04370856bab453964915aef500e4e6e5a8ba9a15102" exitCode=0 Oct 30 21:58:29.809064 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:29.808952 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" event={"ID":"d01f96a5-59c1-4e35-8c31-d4a01e190821","Type":"ContainerDied","Data":"189cfecbf89b0d00670af04370856bab453964915aef500e4e6e5a8ba9a15102"} Oct 30 21:58:30.371054 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:30.371029 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:30.371464 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:30.371448 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:30.372236 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:30.372218 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 21:58:30.978824 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:30.978802 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:31.051006 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.050981 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01f96a5-59c1-4e35-8c31-d4a01e190821-kube-api-access-8ljwg\" (UniqueName: \"kubernetes.io/projected/d01f96a5-59c1-4e35-8c31-d4a01e190821-kube-api-access-8ljwg\") pod \"d01f96a5-59c1-4e35-8c31-d4a01e190821\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " Oct 30 21:58:31.051131 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.051050 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-xks-ca\") pod \"d01f96a5-59c1-4e35-8c31-d4a01e190821\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " Oct 30 21:58:31.051177 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.051140 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-openshift-ca\") pod \"d01f96a5-59c1-4e35-8c31-d4a01e190821\" (UID: \"d01f96a5-59c1-4e35-8c31-d4a01e190821\") " Oct 30 21:58:31.051477 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.051454 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-openshift-ca" pod "d01f96a5-59c1-4e35-8c31-d4a01e190821" (UID: "d01f96a5-59c1-4e35-8c31-d4a01e190821"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:31.053280 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.053258 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01f96a5-59c1-4e35-8c31-d4a01e190821-kube-api-access-8ljwg" pod "d01f96a5-59c1-4e35-8c31-d4a01e190821" (UID: "d01f96a5-59c1-4e35-8c31-d4a01e190821"). InnerVolumeSpecName "kube-api-access-8ljwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:31.053280 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.053264 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-xks-ca" pod "d01f96a5-59c1-4e35-8c31-d4a01e190821" (UID: "d01f96a5-59c1-4e35-8c31-d4a01e190821"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:31.152649 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.152575 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljwg\" (UniqueName: \"kubernetes.io/projected/d01f96a5-59c1-4e35-8c31-d4a01e190821-kube-api-access-8ljwg\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:31.152649 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.152612 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:31.152649 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.152622 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d01f96a5-59c1-4e35-8c31-d4a01e190821-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:31.822273 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.822244 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" Oct 30 21:58:31.822434 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.822277 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2" event={"ID":"d01f96a5-59c1-4e35-8c31-d4a01e190821","Type":"ContainerDied","Data":"85522cd2f2bbbf30183c1af3ffe0f09f0f3accbef0df31507135955aa8e51b9e"} Oct 30 21:58:31.822434 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:31.822328 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85522cd2f2bbbf30183c1af3ffe0f09f0f3accbef0df31507135955aa8e51b9e" Oct 30 21:58:36.121428 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.121399 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq"] Oct 30 21:58:36.121896 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.121860 4933 cpu_manager.go:414] "containerMap: removing container" podUID="d01f96a5-59c1-4e35-8c31-d4a01e190821" containerName="revoke-keys" Oct 30 21:58:36.121896 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.121872 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01f96a5-59c1-4e35-8c31-d4a01e190821" containerName="revoke-keys" Oct 30 21:58:36.122019 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.121968 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01f96a5-59c1-4e35-8c31-d4a01e190821" containerName="revoke-keys" Oct 30 21:58:36.125698 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.125679 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.180162 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.180133 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:36.180271 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.180195 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:36.230700 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.230673 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:58:36.233907 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.233882 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq"] Oct 30 21:58:36.287179 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.287160 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:36.301637 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.301608 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.301742 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.301655 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.301798 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.301759 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhft\" (UniqueName: \"kubernetes.io/projected/735d1747-398e-4fee-a712-24a4a4b4a7b1-kube-api-access-2zhft\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.402604 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.402553 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhft\" (UniqueName: \"kubernetes.io/projected/735d1747-398e-4fee-a712-24a4a4b4a7b1-kube-api-access-2zhft\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.402688 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.402612 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.402688 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.402646 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.403206 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.403185 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.405073 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.405049 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.460539 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.460511 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhft\" (UniqueName: \"kubernetes.io/projected/735d1747-398e-4fee-a712-24a4a4b4a7b1-kube-api-access-2zhft\") pod \"maas-api-revoke-sub-b91e7cec-csshq\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:36.739190 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:36.739118 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:37.122950 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.122920 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq"] Oct 30 21:58:37.123336 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:37.123162 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735d1747_398e_4fee_a712_24a4a4b4a7b1.slice/crio-69742319d56aa6f8488b2c1f0bc9da1cd452df13d09c887ba27c3e7b13117b60 WatchSource:0}: Error finding container 69742319d56aa6f8488b2c1f0bc9da1cd452df13d09c887ba27c3e7b13117b60: Status 404 returned error can't find the container with id 69742319d56aa6f8488b2c1f0bc9da1cd452df13d09c887ba27c3e7b13117b60 Oct 30 21:58:37.688512 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.687985 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76"] Oct 30 21:58:37.693552 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.692750 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.746270 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.746245 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-29cbad-openshift-default-dockercfg-c5s8g\"" Oct 30 21:58:37.800395 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.800357 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76"] Oct 30 21:58:37.817701 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.817659 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlbh\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-kube-api-access-5nlbh\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.817868 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.817713 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.817868 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.817846 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.817956 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.817936 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.818000 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.817966 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.818000 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.817997 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.818162 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.818142 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.818227 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.818210 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.818293 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.818273 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.867761 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.867736 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" event={"ID":"735d1747-398e-4fee-a712-24a4a4b4a7b1","Type":"ContainerStarted","Data":"6b352bff0528fbd7f504b91713cdeed67331031c74262a8706d3f8de62034ea5"} Oct 30 21:58:37.867927 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.867794 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" event={"ID":"735d1747-398e-4fee-a712-24a4a4b4a7b1","Type":"ContainerStarted","Data":"69742319d56aa6f8488b2c1f0bc9da1cd452df13d09c887ba27c3e7b13117b60"} Oct 30 21:58:37.919023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.918991 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919128 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919066 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlbh\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-kube-api-access-5nlbh\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919128 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919095 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919204 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919130 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919204 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919176 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919282 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919205 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919282 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919253 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919377 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919307 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919377 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919365 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919565 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919468 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919565 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919552 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919771 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919751 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.919886 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.919865 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.920520 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.920472 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.921804 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.921781 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.921883 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.921826 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.977660 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.977606 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlbh\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-kube-api-access-5nlbh\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.977775 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.977757 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:37.985588 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:37.985541 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" podStartSLOduration=1.9855247440000001 podStartE2EDuration="1.985524744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:37.984222171 +0000 UTC m=+966.665372061" watchObservedRunningTime="2026-10-30 21:58:37.985524744 +0000 UTC m=+966.666674638" Oct 30 21:58:38.009455 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.009431 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:38.438673 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.438642 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:38.438955 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.438708 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:38.438955 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.438738 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:58:38.439388 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.439291 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76"] Oct 30 21:58:38.876746 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.876710 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" event={"ID":"b2bf740a-f41e-46a3-8fcf-7b94cf50919f","Type":"ContainerStarted","Data":"3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65"} Oct 30 21:58:38.876910 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.876771 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" event={"ID":"b2bf740a-f41e-46a3-8fcf-7b94cf50919f","Type":"ContainerStarted","Data":"3052b5ff3810b34712b09970122529478ac9632cddcdeb767b46a247b5b02937"} Oct 30 21:58:38.878393 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.878364 4933 generic.go:356] "Generic (PLEG): container finished" podID="735d1747-398e-4fee-a712-24a4a4b4a7b1" containerID="6b352bff0528fbd7f504b91713cdeed67331031c74262a8706d3f8de62034ea5" exitCode=0 Oct 30 21:58:38.878548 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:38.878443 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" event={"ID":"735d1747-398e-4fee-a712-24a4a4b4a7b1","Type":"ContainerDied","Data":"6b352bff0528fbd7f504b91713cdeed67331031c74262a8706d3f8de62034ea5"} Oct 30 21:58:39.009370 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:39.009306 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podStartSLOduration=2.009292241 podStartE2EDuration="2.009292241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:38.995723809 +0000 UTC m=+967.676873703" watchObservedRunningTime="2026-10-30 21:58:39.009292241 +0000 UTC m=+967.690442133" Oct 30 21:58:39.009722 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:39.009703 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:39.011435 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:39.011415 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:58:39.011499 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:39.011462 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:58:40.010179 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.010148 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:58:40.010677 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.010208 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:58:40.066804 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.066775 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:40.162952 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.162861 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-xks-ca\") pod \"735d1747-398e-4fee-a712-24a4a4b4a7b1\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " Oct 30 21:58:40.163127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.162982 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-openshift-ca\") pod \"735d1747-398e-4fee-a712-24a4a4b4a7b1\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " Oct 30 21:58:40.163328 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.163305 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-openshift-ca" pod "735d1747-398e-4fee-a712-24a4a4b4a7b1" (UID: "735d1747-398e-4fee-a712-24a4a4b4a7b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:40.165193 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.165162 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-xks-ca" pod "735d1747-398e-4fee-a712-24a4a4b4a7b1" (UID: "735d1747-398e-4fee-a712-24a4a4b4a7b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:40.264321 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.264294 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735d1747-398e-4fee-a712-24a4a4b4a7b1-kube-api-access-2zhft\" (UniqueName: \"kubernetes.io/projected/735d1747-398e-4fee-a712-24a4a4b4a7b1-kube-api-access-2zhft\") pod \"735d1747-398e-4fee-a712-24a4a4b4a7b1\" (UID: \"735d1747-398e-4fee-a712-24a4a4b4a7b1\") " Oct 30 21:58:40.264633 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.264618 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:40.264633 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.264634 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735d1747-398e-4fee-a712-24a4a4b4a7b1-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:40.266627 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.266600 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735d1747-398e-4fee-a712-24a4a4b4a7b1-kube-api-access-2zhft" pod "735d1747-398e-4fee-a712-24a4a4b4a7b1" (UID: "735d1747-398e-4fee-a712-24a4a4b4a7b1"). InnerVolumeSpecName "kube-api-access-2zhft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:40.366132 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.366093 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zhft\" (UniqueName: \"kubernetes.io/projected/735d1747-398e-4fee-a712-24a4a4b4a7b1-kube-api-access-2zhft\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:40.893821 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.893795 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" Oct 30 21:58:40.893965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.893827 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq" event={"ID":"735d1747-398e-4fee-a712-24a4a4b4a7b1","Type":"ContainerDied","Data":"69742319d56aa6f8488b2c1f0bc9da1cd452df13d09c887ba27c3e7b13117b60"} Oct 30 21:58:40.893965 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:40.893874 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69742319d56aa6f8488b2c1f0bc9da1cd452df13d09c887ba27c3e7b13117b60" Oct 30 21:58:41.010581 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:41.010550 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:58:41.010933 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:41.010605 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:58:42.009927 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:42.009886 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:58:42.010122 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:42.009936 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:58:43.010551 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:43.010520 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:58:43.010928 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:43.010577 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:58:44.009930 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:44.009897 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:58:44.010097 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:44.009951 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:58:45.183041 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:45.183011 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:45.183514 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:45.183471 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:45.184275 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:45.184235 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 21:58:48.471116 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.471085 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8"] Oct 30 21:58:48.471784 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.471575 4933 cpu_manager.go:414] "containerMap: removing container" podUID="735d1747-398e-4fee-a712-24a4a4b4a7b1" containerName="revoke-keys" Oct 30 21:58:48.471784 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.471587 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735d1747-398e-4fee-a712-24a4a4b4a7b1" containerName="revoke-keys" Oct 30 21:58:48.471784 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.471680 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="735d1747-398e-4fee-a712-24a4a4b4a7b1" containerName="revoke-keys" Oct 30 21:58:48.475410 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.475383 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.529329 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.529297 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8"] Oct 30 21:58:48.569384 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.569349 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.569590 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.569426 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.569671 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.569628 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkt56\" (UniqueName: \"kubernetes.io/projected/0261f241-2920-497f-92cd-a728f6ca3a12-kube-api-access-qkt56\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.575424 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.575400 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:48.575662 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.575606 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:48.575662 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.575468 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:58:48.575846 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.575712 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:48.670532 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.670472 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt56\" (UniqueName: \"kubernetes.io/projected/0261f241-2920-497f-92cd-a728f6ca3a12-kube-api-access-qkt56\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.670726 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.670590 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.670726 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.670642 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.671884 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.671856 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.674019 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.673993 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.730558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.730443 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt56\" (UniqueName: \"kubernetes.io/projected/0261f241-2920-497f-92cd-a728f6ca3a12-kube-api-access-qkt56\") pod \"maas-api-revoke-sub-3e630f73-dvkp8\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:48.793906 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:48.793870 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:49.111973 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.111938 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q"] Oct 30 21:58:49.116183 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.116165 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.175685 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.175646 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q"] Oct 30 21:58:49.180305 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.180275 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.180472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.180392 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.180472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.180443 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zpm\" (UniqueName: \"kubernetes.io/projected/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-kube-api-access-j7zpm\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.220314 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.220289 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8"] Oct 30 21:58:49.225399 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:49.225370 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0261f241_2920_497f_92cd_a728f6ca3a12.slice/crio-db774977f8a5260be84f92df682bce0872a59af034fb7e5ec7f043628b89f310 WatchSource:0}: Error finding container db774977f8a5260be84f92df682bce0872a59af034fb7e5ec7f043628b89f310: Status 404 returned error can't find the container with id db774977f8a5260be84f92df682bce0872a59af034fb7e5ec7f043628b89f310 Oct 30 21:58:49.281307 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.281276 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.281456 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.281376 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.281456 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.281421 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zpm\" (UniqueName: \"kubernetes.io/projected/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-kube-api-access-j7zpm\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.282682 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.282615 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.284581 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.284553 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.341548 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.341484 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zpm\" (UniqueName: \"kubernetes.io/projected/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-kube-api-access-j7zpm\") pod \"maas-api-revoke-sub-1386df0b-7vp4q\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.429858 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.429778 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:49.899647 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:49.899616 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a5b995_ad9d_4102_92eb_0e0b5d094cd3.slice/crio-7d7c0667da28c497b3d3356ab561beb5ccfabfc065230b0a090c67cbfc7385b6 WatchSource:0}: Error finding container 7d7c0667da28c497b3d3356ab561beb5ccfabfc065230b0a090c67cbfc7385b6: Status 404 returned error can't find the container with id 7d7c0667da28c497b3d3356ab561beb5ccfabfc065230b0a090c67cbfc7385b6 Oct 30 21:58:49.901442 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.901419 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q"] Oct 30 21:58:49.965669 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.965638 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" event={"ID":"60a5b995-ad9d-4102-92eb-0e0b5d094cd3","Type":"ContainerStarted","Data":"7d7c0667da28c497b3d3356ab561beb5ccfabfc065230b0a090c67cbfc7385b6"} Oct 30 21:58:49.967417 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.967391 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" event={"ID":"0261f241-2920-497f-92cd-a728f6ca3a12","Type":"ContainerStarted","Data":"42f190def8a6afd49645eba99b1eed472686e72cb86e7bc2b89293fa45527604"} Oct 30 21:58:49.967529 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:49.967440 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" event={"ID":"0261f241-2920-497f-92cd-a728f6ca3a12","Type":"ContainerStarted","Data":"db774977f8a5260be84f92df682bce0872a59af034fb7e5ec7f043628b89f310"} Oct 30 21:58:50.098944 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:50.092253 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" podStartSLOduration=2.092224028 podStartE2EDuration="2.092224028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:50.088407023 +0000 UTC m=+978.769556919" watchObservedRunningTime="2026-10-30 21:58:50.092224028 +0000 UTC m=+978.773373921" Oct 30 21:58:50.975152 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:50.975111 4933 generic.go:356] "Generic (PLEG): container finished" podID="60a5b995-ad9d-4102-92eb-0e0b5d094cd3" containerID="ca2cdff939eb43dae6a445f50a190417e7ef7a6ae3af46524bc1fd1d70c40ff2" exitCode=0 Oct 30 21:58:50.975600 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:50.975213 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" event={"ID":"60a5b995-ad9d-4102-92eb-0e0b5d094cd3","Type":"ContainerDied","Data":"ca2cdff939eb43dae6a445f50a190417e7ef7a6ae3af46524bc1fd1d70c40ff2"} Oct 30 21:58:50.977282 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:50.977242 4933 generic.go:356] "Generic (PLEG): container finished" podID="0261f241-2920-497f-92cd-a728f6ca3a12" containerID="42f190def8a6afd49645eba99b1eed472686e72cb86e7bc2b89293fa45527604" exitCode=0 Oct 30 21:58:50.977412 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:50.977300 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" event={"ID":"0261f241-2920-497f-92cd-a728f6ca3a12","Type":"ContainerDied","Data":"42f190def8a6afd49645eba99b1eed472686e72cb86e7bc2b89293fa45527604"} Oct 30 21:58:52.155532 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.155506 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:52.182502 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.182461 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:52.213534 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.213507 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-openshift-ca\") pod \"0261f241-2920-497f-92cd-a728f6ca3a12\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " Oct 30 21:58:52.213645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.213577 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-xks-ca\") pod \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " Oct 30 21:58:52.213645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.213609 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-xks-ca\") pod \"0261f241-2920-497f-92cd-a728f6ca3a12\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " Oct 30 21:58:52.213645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.213637 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-openshift-ca\") pod \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " Oct 30 21:58:52.213784 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.213744 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-kube-api-access-j7zpm\" (UniqueName: \"kubernetes.io/projected/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-kube-api-access-j7zpm\") pod \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\" (UID: \"60a5b995-ad9d-4102-92eb-0e0b5d094cd3\") " Oct 30 21:58:52.213842 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.213794 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-openshift-ca" pod "0261f241-2920-497f-92cd-a728f6ca3a12" (UID: "0261f241-2920-497f-92cd-a728f6ca3a12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:52.213899 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.213867 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0261f241-2920-497f-92cd-a728f6ca3a12-kube-api-access-qkt56\" (UniqueName: \"kubernetes.io/projected/0261f241-2920-497f-92cd-a728f6ca3a12-kube-api-access-qkt56\") pod \"0261f241-2920-497f-92cd-a728f6ca3a12\" (UID: \"0261f241-2920-497f-92cd-a728f6ca3a12\") " Oct 30 21:58:52.214081 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.214056 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-openshift-ca" pod "60a5b995-ad9d-4102-92eb-0e0b5d094cd3" (UID: "60a5b995-ad9d-4102-92eb-0e0b5d094cd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:52.214319 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.214269 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:52.214319 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.214294 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:52.215912 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.215883 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-xks-ca" pod "60a5b995-ad9d-4102-92eb-0e0b5d094cd3" (UID: "60a5b995-ad9d-4102-92eb-0e0b5d094cd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:52.216020 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.215945 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-xks-ca" pod "0261f241-2920-497f-92cd-a728f6ca3a12" (UID: "0261f241-2920-497f-92cd-a728f6ca3a12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:52.216088 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.216045 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-kube-api-access-j7zpm" pod "60a5b995-ad9d-4102-92eb-0e0b5d094cd3" (UID: "60a5b995-ad9d-4102-92eb-0e0b5d094cd3"). InnerVolumeSpecName "kube-api-access-j7zpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:52.216171 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.216143 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0261f241-2920-497f-92cd-a728f6ca3a12-kube-api-access-qkt56" pod "0261f241-2920-497f-92cd-a728f6ca3a12" (UID: "0261f241-2920-497f-92cd-a728f6ca3a12"). InnerVolumeSpecName "kube-api-access-qkt56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:52.315696 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.315664 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkt56\" (UniqueName: \"kubernetes.io/projected/0261f241-2920-497f-92cd-a728f6ca3a12-kube-api-access-qkt56\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:52.315696 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.315694 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:52.315879 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.315709 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0261f241-2920-497f-92cd-a728f6ca3a12-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:52.315879 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.315721 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zpm\" (UniqueName: \"kubernetes.io/projected/60a5b995-ad9d-4102-92eb-0e0b5d094cd3-kube-api-access-j7zpm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:58:52.992686 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.992657 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" Oct 30 21:58:52.992873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.992658 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q" event={"ID":"60a5b995-ad9d-4102-92eb-0e0b5d094cd3","Type":"ContainerDied","Data":"7d7c0667da28c497b3d3356ab561beb5ccfabfc065230b0a090c67cbfc7385b6"} Oct 30 21:58:52.992873 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.992792 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d7c0667da28c497b3d3356ab561beb5ccfabfc065230b0a090c67cbfc7385b6" Oct 30 21:58:52.994054 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.994029 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" event={"ID":"0261f241-2920-497f-92cd-a728f6ca3a12","Type":"ContainerDied","Data":"db774977f8a5260be84f92df682bce0872a59af034fb7e5ec7f043628b89f310"} Oct 30 21:58:52.994157 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.994080 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db774977f8a5260be84f92df682bce0872a59af034fb7e5ec7f043628b89f310" Oct 30 21:58:52.994157 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:52.994041 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8" Oct 30 21:58:57.873410 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.873381 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl"] Oct 30 21:58:57.873890 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.873876 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0261f241-2920-497f-92cd-a728f6ca3a12" containerName="revoke-keys" Oct 30 21:58:57.873935 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.873892 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0261f241-2920-497f-92cd-a728f6ca3a12" containerName="revoke-keys" Oct 30 21:58:57.873935 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.873918 4933 cpu_manager.go:414] "containerMap: removing container" podUID="60a5b995-ad9d-4102-92eb-0e0b5d094cd3" containerName="revoke-keys" Oct 30 21:58:57.873935 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.873924 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a5b995-ad9d-4102-92eb-0e0b5d094cd3" containerName="revoke-keys" Oct 30 21:58:57.874029 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.874024 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0261f241-2920-497f-92cd-a728f6ca3a12" containerName="revoke-keys" Oct 30 21:58:57.874061 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.874037 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a5b995-ad9d-4102-92eb-0e0b5d094cd3" containerName="revoke-keys" Oct 30 21:58:57.878458 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.878440 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:57.933564 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.933541 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl"] Oct 30 21:58:57.973501 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.973466 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:57.973648 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.973618 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:57.973735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.973680 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7688s\" (UniqueName: \"kubernetes.io/projected/e31c1bf5-c985-4791-99b0-d617606f2cc7-kube-api-access-7688s\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:57.980276 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.980258 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:57.981717 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.981699 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:57.981799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.981721 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:58:57.981799 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:57.981754 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:58.074838 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.074698 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:58.074838 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.074774 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7688s\" (UniqueName: \"kubernetes.io/projected/e31c1bf5-c985-4791-99b0-d617606f2cc7-kube-api-access-7688s\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:58.075104 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.074855 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:58.075571 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.075550 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:58.077765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.077737 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:58.132507 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.132437 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7688s\" (UniqueName: \"kubernetes.io/projected/e31c1bf5-c985-4791-99b0-d617606f2cc7-kube-api-access-7688s\") pod \"maas-api-revoke-sub-3d917d11-xx2gl\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:58.191357 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.191324 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:58:58.544309 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:58:58.544240 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:58:58.570263 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:58:58.570232 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31c1bf5_c985_4791_99b0_d617606f2cc7.slice/crio-5b1b6c3529b335c06903d4db337b512f5fcb4d5b42b9491b5fba45dd95b36498 WatchSource:0}: Error finding container 5b1b6c3529b335c06903d4db337b512f5fcb4d5b42b9491b5fba45dd95b36498: Status 404 returned error can't find the container with id 5b1b6c3529b335c06903d4db337b512f5fcb4d5b42b9491b5fba45dd95b36498 Oct 30 21:58:58.570412 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:58.570393 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl"] Oct 30 21:58:59.040996 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:59.040959 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" event={"ID":"e31c1bf5-c985-4791-99b0-d617606f2cc7","Type":"ContainerStarted","Data":"6c3fbaffaaa3616c6cf1791b96183089e8ec97d77b6013724a0745da24b1ee34"} Oct 30 21:58:59.041369 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:59.041026 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" event={"ID":"e31c1bf5-c985-4791-99b0-d617606f2cc7","Type":"ContainerStarted","Data":"5b1b6c3529b335c06903d4db337b512f5fcb4d5b42b9491b5fba45dd95b36498"} Oct 30 21:58:59.157314 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:58:59.157215 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" podStartSLOduration=2.157202162 podStartE2EDuration="2.157202162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:59.154392403 +0000 UTC m=+987.835542298" watchObservedRunningTime="2026-10-30 21:58:59.157202162 +0000 UTC m=+987.838352055" Oct 30 21:59:00.049035 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:00.049003 4933 generic.go:356] "Generic (PLEG): container finished" podID="e31c1bf5-c985-4791-99b0-d617606f2cc7" containerID="6c3fbaffaaa3616c6cf1791b96183089e8ec97d77b6013724a0745da24b1ee34" exitCode=0 Oct 30 21:59:00.049412 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:00.049089 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" event={"ID":"e31c1bf5-c985-4791-99b0-d617606f2cc7","Type":"ContainerDied","Data":"6c3fbaffaaa3616c6cf1791b96183089e8ec97d77b6013724a0745da24b1ee34"} Oct 30 21:59:01.205631 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.205608 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:59:01.307567 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.307456 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-openshift-ca\") pod \"e31c1bf5-c985-4791-99b0-d617606f2cc7\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " Oct 30 21:59:01.307731 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.307628 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31c1bf5-c985-4791-99b0-d617606f2cc7-kube-api-access-7688s\" (UniqueName: \"kubernetes.io/projected/e31c1bf5-c985-4791-99b0-d617606f2cc7-kube-api-access-7688s\") pod \"e31c1bf5-c985-4791-99b0-d617606f2cc7\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " Oct 30 21:59:01.307800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.307725 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-xks-ca\") pod \"e31c1bf5-c985-4791-99b0-d617606f2cc7\" (UID: \"e31c1bf5-c985-4791-99b0-d617606f2cc7\") " Oct 30 21:59:01.307855 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.307803 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-openshift-ca" pod "e31c1bf5-c985-4791-99b0-d617606f2cc7" (UID: "e31c1bf5-c985-4791-99b0-d617606f2cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:01.308066 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.308048 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:01.309887 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.309865 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-xks-ca" pod "e31c1bf5-c985-4791-99b0-d617606f2cc7" (UID: "e31c1bf5-c985-4791-99b0-d617606f2cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:01.309978 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.309931 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31c1bf5-c985-4791-99b0-d617606f2cc7-kube-api-access-7688s" pod "e31c1bf5-c985-4791-99b0-d617606f2cc7" (UID: "e31c1bf5-c985-4791-99b0-d617606f2cc7"). InnerVolumeSpecName "kube-api-access-7688s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:01.409327 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.409298 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e31c1bf5-c985-4791-99b0-d617606f2cc7-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:01.409327 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:01.409324 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7688s\" (UniqueName: \"kubernetes.io/projected/e31c1bf5-c985-4791-99b0-d617606f2cc7-kube-api-access-7688s\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:02.063095 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.063070 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" Oct 30 21:59:02.063270 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.063064 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl" event={"ID":"e31c1bf5-c985-4791-99b0-d617606f2cc7","Type":"ContainerDied","Data":"5b1b6c3529b335c06903d4db337b512f5fcb4d5b42b9491b5fba45dd95b36498"} Oct 30 21:59:02.063270 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.063188 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b1b6c3529b335c06903d4db337b512f5fcb4d5b42b9491b5fba45dd95b36498" Oct 30 21:59:02.875353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.875312 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x"] Oct 30 21:59:02.875875 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.875860 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e31c1bf5-c985-4791-99b0-d617606f2cc7" containerName="revoke-keys" Oct 30 21:59:02.875939 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.875876 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31c1bf5-c985-4791-99b0-d617606f2cc7" containerName="revoke-keys" Oct 30 21:59:02.875981 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.875971 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31c1bf5-c985-4791-99b0-d617606f2cc7" containerName="revoke-keys" Oct 30 21:59:02.879432 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.879411 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.925823 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.925799 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.925942 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.925842 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.925942 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.925863 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.926017 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.925993 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.926050 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.926028 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.926153 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.926138 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.926196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.926186 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec7a677-1d2b-416f-8de9-0a65229fd10d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.926293 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.926273 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:02.926330 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:02.926313 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ld8d\" (UniqueName: \"kubernetes.io/projected/fec7a677-1d2b-416f-8de9-0a65229fd10d-kube-api-access-2ld8d\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027312 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027286 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027442 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027314 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027442 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027382 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027442 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027424 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec7a677-1d2b-416f-8de9-0a65229fd10d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027633 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027608 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027698 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027663 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ld8d\" (UniqueName: \"kubernetes.io/projected/fec7a677-1d2b-416f-8de9-0a65229fd10d-kube-api-access-2ld8d\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027757 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027744 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027806 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027796 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027861 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027826 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027861 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027846 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.027962 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.027891 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.028023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.028000 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.028174 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.028154 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.028356 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.028336 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fec7a677-1d2b-416f-8de9-0a65229fd10d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.029913 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.029894 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.030438 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.030422 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.082303 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.082277 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x"] Oct 30 21:59:03.089614 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.089584 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ld8d\" (UniqueName: \"kubernetes.io/projected/fec7a677-1d2b-416f-8de9-0a65229fd10d-kube-api-access-2ld8d\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.092315 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.092292 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fec7a677-1d2b-416f-8de9-0a65229fd10d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-2877x\" (UID: \"fec7a677-1d2b-416f-8de9-0a65229fd10d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.193145 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.193073 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:03.634148 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.634122 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x"] Oct 30 21:59:03.634563 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:03.634536 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec7a677_1d2b_416f_8de9_0a65229fd10d.slice/crio-42bf6cef64237b019e63d3072a0a7682af14cbd53bdd033fc4598bd85ad7b35e WatchSource:0}: Error finding container 42bf6cef64237b019e63d3072a0a7682af14cbd53bdd033fc4598bd85ad7b35e: Status 404 returned error can't find the container with id 42bf6cef64237b019e63d3072a0a7682af14cbd53bdd033fc4598bd85ad7b35e Oct 30 21:59:03.637467 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.637431 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:03.637583 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.637518 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:03.637583 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.637553 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:03.910218 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.910130 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt"] Oct 30 21:59:03.914820 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.914798 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:03.961991 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.961965 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt"] Oct 30 21:59:03.968228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.968019 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:03.968228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.968092 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:03.968228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.968183 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:59:03.968228 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:03.968229 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:04.040515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.040105 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8wj\" (UniqueName: \"kubernetes.io/projected/19c18b75-ce39-4fc0-a760-051d66ae16ae-kube-api-access-wv8wj\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.040515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.040267 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.040515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.040339 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.083012 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.082982 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" event={"ID":"fec7a677-1d2b-416f-8de9-0a65229fd10d","Type":"ContainerStarted","Data":"6bccd2655b77a2b7a23e190475f7499e4d6d3f08e5bf38fba234c43229fd9599"} Oct 30 21:59:04.083201 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.083031 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" event={"ID":"fec7a677-1d2b-416f-8de9-0a65229fd10d","Type":"ContainerStarted","Data":"42bf6cef64237b019e63d3072a0a7682af14cbd53bdd033fc4598bd85ad7b35e"} Oct 30 21:59:04.141262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.141223 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8wj\" (UniqueName: \"kubernetes.io/projected/19c18b75-ce39-4fc0-a760-051d66ae16ae-kube-api-access-wv8wj\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.141462 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.141378 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.141578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.141540 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.142169 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.142141 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.144422 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.144395 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.207198 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.207109 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:04.211689 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.211199 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8wj\" (UniqueName: \"kubernetes.io/projected/19c18b75-ce39-4fc0-a760-051d66ae16ae-kube-api-access-wv8wj\") pod \"maas-api-revoke-sub-46489f2d-f4gdt\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.215633 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.215599 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:04.215782 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.215657 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:04.225684 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.225623 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podStartSLOduration=2.225609235 podStartE2EDuration="2.225609235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:04.22457925 +0000 UTC m=+992.905729144" watchObservedRunningTime="2026-10-30 21:59:04.225609235 +0000 UTC m=+992.906759128" Oct 30 21:59:04.234512 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.230746 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:04.483341 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.483267 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc"] Oct 30 21:59:04.488476 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.488451 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.565277 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.565242 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc"] Oct 30 21:59:04.578023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.577408 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.578023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.577470 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.578023 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.577522 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgsl\" (UniqueName: \"kubernetes.io/projected/e9189e58-8445-4bc0-936f-a2692d037906-kube-api-access-5rgsl\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.661591 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.661568 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6"] Oct 30 21:59:04.664127 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:04.664098 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c18b75_ce39_4fc0_a760_051d66ae16ae.slice/crio-94542834ceb0c41f73a6a68d803344bb3075c5562e9809136eed2971f2977192 WatchSource:0}: Error finding container 94542834ceb0c41f73a6a68d803344bb3075c5562e9809136eed2971f2977192: Status 404 returned error can't find the container with id 94542834ceb0c41f73a6a68d803344bb3075c5562e9809136eed2971f2977192 Oct 30 21:59:04.665813 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.665713 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt"] Oct 30 21:59:04.665813 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.665754 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.678134 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.678114 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.678221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.678164 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.678221 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.678201 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgsl\" (UniqueName: \"kubernetes.io/projected/e9189e58-8445-4bc0-936f-a2692d037906-kube-api-access-5rgsl\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.678857 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.678833 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.681196 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.681171 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.703684 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.700071 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6"] Oct 30 21:59:04.779611 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.779573 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.779767 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.779687 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmjn\" (UniqueName: \"kubernetes.io/projected/d48c73ad-e333-4fc6-b215-1c087df592d6-kube-api-access-lzmjn\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.779872 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.779854 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.850514 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.850220 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgsl\" (UniqueName: \"kubernetes.io/projected/e9189e58-8445-4bc0-936f-a2692d037906-kube-api-access-5rgsl\") pod \"maas-api-revoke-sub-9202767f-lllfc\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:04.881239 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.881198 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.881381 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.881305 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.881381 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.881354 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmjn\" (UniqueName: \"kubernetes.io/projected/d48c73ad-e333-4fc6-b215-1c087df592d6-kube-api-access-lzmjn\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.882168 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.882141 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.887373 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.885829 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.954538 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.952068 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmjn\" (UniqueName: \"kubernetes.io/projected/d48c73ad-e333-4fc6-b215-1c087df592d6-kube-api-access-lzmjn\") pod \"maas-api-revoke-sub-e8adb9c8-zqtc6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:04.982144 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:04.982117 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:05.112980 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.111742 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:05.120158 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.120128 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" event={"ID":"19c18b75-ce39-4fc0-a760-051d66ae16ae","Type":"ContainerStarted","Data":"5bc4b04d444f3da852cc2ddd845c7d6af64062b54e8a961f8fc4d7103b88e91c"} Oct 30 21:59:05.120567 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.120484 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" event={"ID":"19c18b75-ce39-4fc0-a760-051d66ae16ae","Type":"ContainerStarted","Data":"94542834ceb0c41f73a6a68d803344bb3075c5562e9809136eed2971f2977192"} Oct 30 21:59:05.199885 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.199855 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:05.200033 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.199915 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:05.276516 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.259438 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" podStartSLOduration=2.25941788 podStartE2EDuration="2.25941788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:05.251848255 +0000 UTC m=+993.932998151" watchObservedRunningTime="2026-10-30 21:59:05.25941788 +0000 UTC m=+993.940567775" Oct 30 21:59:05.427101 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.427072 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6"] Oct 30 21:59:05.427392 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:05.427366 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48c73ad_e333_4fc6_b215_1c087df592d6.slice/crio-dde4671543764e640fcb51c4b72b43d8fe3e3e3fa9a8ba9c2213ee1e3dcfd59b WatchSource:0}: Error finding container dde4671543764e640fcb51c4b72b43d8fe3e3e3fa9a8ba9c2213ee1e3dcfd59b: Status 404 returned error can't find the container with id dde4671543764e640fcb51c4b72b43d8fe3e3e3fa9a8ba9c2213ee1e3dcfd59b Oct 30 21:59:05.620630 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:05.620599 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc"] Oct 30 21:59:06.127181 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.127146 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" event={"ID":"e9189e58-8445-4bc0-936f-a2692d037906","Type":"ContainerStarted","Data":"61563de3a85c34a1dec16a44a65e858871965f1ab0572cb4630dd3851434cf04"} Oct 30 21:59:06.127574 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.127204 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" event={"ID":"e9189e58-8445-4bc0-936f-a2692d037906","Type":"ContainerStarted","Data":"c824a72a500d5985bfe9128108ad7a66837987e657658adafd8486fcc509958c"} Oct 30 21:59:06.129006 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.128979 4933 generic.go:356] "Generic (PLEG): container finished" podID="19c18b75-ce39-4fc0-a760-051d66ae16ae" containerID="5bc4b04d444f3da852cc2ddd845c7d6af64062b54e8a961f8fc4d7103b88e91c" exitCode=0 Oct 30 21:59:06.129114 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.129050 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" event={"ID":"19c18b75-ce39-4fc0-a760-051d66ae16ae","Type":"ContainerDied","Data":"5bc4b04d444f3da852cc2ddd845c7d6af64062b54e8a961f8fc4d7103b88e91c"} Oct 30 21:59:06.145513 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.139878 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" event={"ID":"d48c73ad-e333-4fc6-b215-1c087df592d6","Type":"ContainerStarted","Data":"9802b88fd15bea20641ee04b387b7dadfbff00b2bf1c94c5ca756f22cb45f390"} Oct 30 21:59:06.145513 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.139923 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" event={"ID":"d48c73ad-e333-4fc6-b215-1c087df592d6","Type":"ContainerStarted","Data":"dde4671543764e640fcb51c4b72b43d8fe3e3e3fa9a8ba9c2213ee1e3dcfd59b"} Oct 30 21:59:06.194870 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.194844 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:06.195011 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.194896 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:06.254516 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.249276 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" podStartSLOduration=2.249257091 podStartE2EDuration="2.249257091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:06.246923424 +0000 UTC m=+994.928073319" watchObservedRunningTime="2026-10-30 21:59:06.249257091 +0000 UTC m=+994.930406985" Oct 30 21:59:06.477431 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:06.477328 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" podStartSLOduration=2.477315114 podStartE2EDuration="2.477315114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:06.475971473 +0000 UTC m=+995.157121368" watchObservedRunningTime="2026-10-30 21:59:06.477315114 +0000 UTC m=+995.158465030" Oct 30 21:59:07.165350 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.165314 4933 generic.go:356] "Generic (PLEG): container finished" podID="e9189e58-8445-4bc0-936f-a2692d037906" containerID="61563de3a85c34a1dec16a44a65e858871965f1ab0572cb4630dd3851434cf04" exitCode=0 Oct 30 21:59:07.165832 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.165402 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" event={"ID":"e9189e58-8445-4bc0-936f-a2692d037906","Type":"ContainerDied","Data":"61563de3a85c34a1dec16a44a65e858871965f1ab0572cb4630dd3851434cf04"} Oct 30 21:59:07.167227 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.167204 4933 generic.go:356] "Generic (PLEG): container finished" podID="d48c73ad-e333-4fc6-b215-1c087df592d6" containerID="9802b88fd15bea20641ee04b387b7dadfbff00b2bf1c94c5ca756f22cb45f390" exitCode=0 Oct 30 21:59:07.167354 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.167253 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" event={"ID":"d48c73ad-e333-4fc6-b215-1c087df592d6","Type":"ContainerDied","Data":"9802b88fd15bea20641ee04b387b7dadfbff00b2bf1c94c5ca756f22cb45f390"} Oct 30 21:59:07.193932 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.193902 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:07.194128 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.194105 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:07.350518 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.348647 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:07.410014 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.409982 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-openshift-ca\") pod \"19c18b75-ce39-4fc0-a760-051d66ae16ae\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " Oct 30 21:59:07.410162 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.410097 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c18b75-ce39-4fc0-a760-051d66ae16ae-kube-api-access-wv8wj\" (UniqueName: \"kubernetes.io/projected/19c18b75-ce39-4fc0-a760-051d66ae16ae-kube-api-access-wv8wj\") pod \"19c18b75-ce39-4fc0-a760-051d66ae16ae\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " Oct 30 21:59:07.410206 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.410173 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-xks-ca\") pod \"19c18b75-ce39-4fc0-a760-051d66ae16ae\" (UID: \"19c18b75-ce39-4fc0-a760-051d66ae16ae\") " Oct 30 21:59:07.410315 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.410293 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-openshift-ca" pod "19c18b75-ce39-4fc0-a760-051d66ae16ae" (UID: "19c18b75-ce39-4fc0-a760-051d66ae16ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:07.410505 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.410471 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:07.412423 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.412399 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c18b75-ce39-4fc0-a760-051d66ae16ae-kube-api-access-wv8wj" pod "19c18b75-ce39-4fc0-a760-051d66ae16ae" (UID: "19c18b75-ce39-4fc0-a760-051d66ae16ae"). InnerVolumeSpecName "kube-api-access-wv8wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:07.412423 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.412409 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-xks-ca" pod "19c18b75-ce39-4fc0-a760-051d66ae16ae" (UID: "19c18b75-ce39-4fc0-a760-051d66ae16ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:07.511831 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.511798 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv8wj\" (UniqueName: \"kubernetes.io/projected/19c18b75-ce39-4fc0-a760-051d66ae16ae-kube-api-access-wv8wj\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:07.511831 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:07.511825 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19c18b75-ce39-4fc0-a760-051d66ae16ae-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:08.176310 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.176277 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" Oct 30 21:59:08.176822 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.176315 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt" event={"ID":"19c18b75-ce39-4fc0-a760-051d66ae16ae","Type":"ContainerDied","Data":"94542834ceb0c41f73a6a68d803344bb3075c5562e9809136eed2971f2977192"} Oct 30 21:59:08.176822 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.176399 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94542834ceb0c41f73a6a68d803344bb3075c5562e9809136eed2971f2977192" Oct 30 21:59:08.194371 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.194338 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:08.194521 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.194395 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:08.419615 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.419595 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:08.422616 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.422600 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:08.525519 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.525470 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-xks-ca\") pod \"e9189e58-8445-4bc0-936f-a2692d037906\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " Oct 30 21:59:08.525655 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.525543 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-xks-ca\") pod \"d48c73ad-e333-4fc6-b215-1c087df592d6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " Oct 30 21:59:08.525655 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.525592 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-openshift-ca\") pod \"e9189e58-8445-4bc0-936f-a2692d037906\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " Oct 30 21:59:08.525734 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.525683 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-openshift-ca\") pod \"d48c73ad-e333-4fc6-b215-1c087df592d6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " Oct 30 21:59:08.525765 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.525750 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48c73ad-e333-4fc6-b215-1c087df592d6-kube-api-access-lzmjn\" (UniqueName: \"kubernetes.io/projected/d48c73ad-e333-4fc6-b215-1c087df592d6-kube-api-access-lzmjn\") pod \"d48c73ad-e333-4fc6-b215-1c087df592d6\" (UID: \"d48c73ad-e333-4fc6-b215-1c087df592d6\") " Oct 30 21:59:08.525809 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.525774 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9189e58-8445-4bc0-936f-a2692d037906-kube-api-access-5rgsl\" (UniqueName: \"kubernetes.io/projected/e9189e58-8445-4bc0-936f-a2692d037906-kube-api-access-5rgsl\") pod \"e9189e58-8445-4bc0-936f-a2692d037906\" (UID: \"e9189e58-8445-4bc0-936f-a2692d037906\") " Oct 30 21:59:08.526339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.525988 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-openshift-ca" pod "e9189e58-8445-4bc0-936f-a2692d037906" (UID: "e9189e58-8445-4bc0-936f-a2692d037906"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:08.526339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.526065 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-openshift-ca" pod "d48c73ad-e333-4fc6-b215-1c087df592d6" (UID: "d48c73ad-e333-4fc6-b215-1c087df592d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:08.526609 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.526408 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:08.526609 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.526431 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:08.528529 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.528484 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-xks-ca" pod "d48c73ad-e333-4fc6-b215-1c087df592d6" (UID: "d48c73ad-e333-4fc6-b215-1c087df592d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:08.528619 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.528559 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-xks-ca" pod "e9189e58-8445-4bc0-936f-a2692d037906" (UID: "e9189e58-8445-4bc0-936f-a2692d037906"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:08.528676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.528625 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48c73ad-e333-4fc6-b215-1c087df592d6-kube-api-access-lzmjn" pod "d48c73ad-e333-4fc6-b215-1c087df592d6" (UID: "d48c73ad-e333-4fc6-b215-1c087df592d6"). InnerVolumeSpecName "kube-api-access-lzmjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:08.528676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.528652 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9189e58-8445-4bc0-936f-a2692d037906-kube-api-access-5rgsl" pod "e9189e58-8445-4bc0-936f-a2692d037906" (UID: "e9189e58-8445-4bc0-936f-a2692d037906"). InnerVolumeSpecName "kube-api-access-5rgsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:08.627170 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.627138 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzmjn\" (UniqueName: \"kubernetes.io/projected/d48c73ad-e333-4fc6-b215-1c087df592d6-kube-api-access-lzmjn\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:08.627170 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.627165 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rgsl\" (UniqueName: \"kubernetes.io/projected/e9189e58-8445-4bc0-936f-a2692d037906-kube-api-access-5rgsl\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:08.627170 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.627175 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9189e58-8445-4bc0-936f-a2692d037906-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:08.627393 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:08.627184 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48c73ad-e333-4fc6-b215-1c087df592d6-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:09.185758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.185730 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" Oct 30 21:59:09.186158 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.185732 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6" event={"ID":"d48c73ad-e333-4fc6-b215-1c087df592d6","Type":"ContainerDied","Data":"dde4671543764e640fcb51c4b72b43d8fe3e3e3fa9a8ba9c2213ee1e3dcfd59b"} Oct 30 21:59:09.186158 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.185859 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde4671543764e640fcb51c4b72b43d8fe3e3e3fa9a8ba9c2213ee1e3dcfd59b" Oct 30 21:59:09.187354 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.187336 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" Oct 30 21:59:09.187354 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.187341 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc" event={"ID":"e9189e58-8445-4bc0-936f-a2692d037906","Type":"ContainerDied","Data":"c824a72a500d5985bfe9128108ad7a66837987e657658adafd8486fcc509958c"} Oct 30 21:59:09.187512 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.187376 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c824a72a500d5985bfe9128108ad7a66837987e657658adafd8486fcc509958c" Oct 30 21:59:09.193641 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.193622 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:09.193723 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:09.193683 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:10.193839 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:10.193806 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:10.194214 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:10.193863 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:11.193796 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:11.193744 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-2877x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 21:59:11.193976 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:11.193818 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" podUID="fec7a677-1d2b-416f-8de9-0a65229fd10d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 21:59:11.415703 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:11.415668 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:59:12.231973 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:12.231940 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:12.232526 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:12.232468 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:12.233617 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:12.233577 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-2877x" Oct 30 21:59:14.440640 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.440603 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r"] Oct 30 21:59:14.441382 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441359 4933 cpu_manager.go:414] "containerMap: removing container" podUID="d48c73ad-e333-4fc6-b215-1c087df592d6" containerName="revoke-keys" Oct 30 21:59:14.441472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441385 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48c73ad-e333-4fc6-b215-1c087df592d6" containerName="revoke-keys" Oct 30 21:59:14.441472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441403 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e9189e58-8445-4bc0-936f-a2692d037906" containerName="revoke-keys" Oct 30 21:59:14.441472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441412 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9189e58-8445-4bc0-936f-a2692d037906" containerName="revoke-keys" Oct 30 21:59:14.441472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441436 4933 cpu_manager.go:414] "containerMap: removing container" podUID="19c18b75-ce39-4fc0-a760-051d66ae16ae" containerName="revoke-keys" Oct 30 21:59:14.441472 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441444 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c18b75-ce39-4fc0-a760-051d66ae16ae" containerName="revoke-keys" Oct 30 21:59:14.441769 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441643 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48c73ad-e333-4fc6-b215-1c087df592d6" containerName="revoke-keys" Oct 30 21:59:14.441769 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441658 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9189e58-8445-4bc0-936f-a2692d037906" containerName="revoke-keys" Oct 30 21:59:14.441769 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.441672 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c18b75-ce39-4fc0-a760-051d66ae16ae" containerName="revoke-keys" Oct 30 21:59:14.446925 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.446903 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.496797 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.496769 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r"] Oct 30 21:59:14.529809 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.529781 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thp6z\" (UniqueName: \"kubernetes.io/projected/9832f67b-f68e-4d85-873a-1f6db6005947-kube-api-access-thp6z\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.530098 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.530066 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.530217 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.530128 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.543127 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.543099 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:14.543224 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.543104 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:14.543374 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.543350 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:14.550162 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.550141 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:59:14.631666 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.631630 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thp6z\" (UniqueName: \"kubernetes.io/projected/9832f67b-f68e-4d85-873a-1f6db6005947-kube-api-access-thp6z\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.631821 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.631736 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.631821 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.631815 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.632674 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.632647 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.634563 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.634539 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.693707 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.693599 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thp6z\" (UniqueName: \"kubernetes.io/projected/9832f67b-f68e-4d85-873a-1f6db6005947-kube-api-access-thp6z\") pod \"maas-api-revoke-sub-35111e14-xv22r\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.707013 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.706979 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t"] Oct 30 21:59:14.713661 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.713634 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.762512 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.762451 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t"] Oct 30 21:59:14.763037 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.763017 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:14.841002 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.838243 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.841002 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.838280 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.841002 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.838350 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx77w\" (UniqueName: \"kubernetes.io/projected/257892d8-a41c-4b4b-96b8-b0985376d220-kube-api-access-zx77w\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.939507 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.939458 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx77w\" (UniqueName: \"kubernetes.io/projected/257892d8-a41c-4b4b-96b8-b0985376d220-kube-api-access-zx77w\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.939693 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.939594 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.939693 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.939637 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.940477 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.940406 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.942766 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.942738 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:14.998291 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:14.998219 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx77w\" (UniqueName: \"kubernetes.io/projected/257892d8-a41c-4b4b-96b8-b0985376d220-kube-api-access-zx77w\") pod \"maas-api-revoke-sub-da03ebac-rnk5t\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:15.029329 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.029306 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:15.175243 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:15.175204 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9832f67b_f68e_4d85_873a_1f6db6005947.slice/crio-246c3dc141b06b69d4b494c605c1cbfa04d0e8fd20c64f5226f04c8d27948b97 WatchSource:0}: Error finding container 246c3dc141b06b69d4b494c605c1cbfa04d0e8fd20c64f5226f04c8d27948b97: Status 404 returned error can't find the container with id 246c3dc141b06b69d4b494c605c1cbfa04d0e8fd20c64f5226f04c8d27948b97 Oct 30 21:59:15.177662 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.177632 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r"] Oct 30 21:59:15.229859 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.229828 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" event={"ID":"9832f67b-f68e-4d85-873a-1f6db6005947","Type":"ContainerStarted","Data":"246c3dc141b06b69d4b494c605c1cbfa04d0e8fd20c64f5226f04c8d27948b97"} Oct 30 21:59:15.315193 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.315168 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl"] Oct 30 21:59:15.318932 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.318910 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.376242 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.376216 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl"] Oct 30 21:59:15.405988 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.405963 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t"] Oct 30 21:59:15.406449 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:15.406425 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257892d8_a41c_4b4b_96b8_b0985376d220.slice/crio-b6b9cfb4daecb36beecb1835bc316fe9605dcb574a44ce411d7fc403cc754db8 WatchSource:0}: Error finding container b6b9cfb4daecb36beecb1835bc316fe9605dcb574a44ce411d7fc403cc754db8: Status 404 returned error can't find the container with id b6b9cfb4daecb36beecb1835bc316fe9605dcb574a44ce411d7fc403cc754db8 Oct 30 21:59:15.445892 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.445863 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.446257 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.446043 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wshsc\" (UniqueName: \"kubernetes.io/projected/eb802b5f-d726-4d17-ad7c-a26a76e9590f-kube-api-access-wshsc\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.446257 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.446175 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.547565 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.547534 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wshsc\" (UniqueName: \"kubernetes.io/projected/eb802b5f-d726-4d17-ad7c-a26a76e9590f-kube-api-access-wshsc\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.547710 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.547656 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.547759 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.547725 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.548324 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.548297 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.550295 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.550269 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.605973 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.605910 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wshsc\" (UniqueName: \"kubernetes.io/projected/eb802b5f-d726-4d17-ad7c-a26a76e9590f-kube-api-access-wshsc\") pod \"maas-api-revoke-sub-b0de3460-vt4pl\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:15.632803 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:15.632781 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:16.009226 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:16.009199 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl"] Oct 30 21:59:16.062531 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:16.062475 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb802b5f_d726_4d17_ad7c_a26a76e9590f.slice/crio-e855c15a8ebfafd3a865f2623ccd915c5a5a048eaec78f1e25cf871f0d0163e6 WatchSource:0}: Error finding container e855c15a8ebfafd3a865f2623ccd915c5a5a048eaec78f1e25cf871f0d0163e6: Status 404 returned error can't find the container with id e855c15a8ebfafd3a865f2623ccd915c5a5a048eaec78f1e25cf871f0d0163e6 Oct 30 21:59:16.248737 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:16.243431 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" event={"ID":"eb802b5f-d726-4d17-ad7c-a26a76e9590f","Type":"ContainerStarted","Data":"e855c15a8ebfafd3a865f2623ccd915c5a5a048eaec78f1e25cf871f0d0163e6"} Oct 30 21:59:16.248737 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:16.245721 4933 generic.go:356] "Generic (PLEG): container finished" podID="9832f67b-f68e-4d85-873a-1f6db6005947" containerID="2cb86c0812988580e3ec53056cc3166b868434ae468bbb1b0be38331e2664d39" exitCode=0 Oct 30 21:59:16.248737 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:16.245798 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" event={"ID":"9832f67b-f68e-4d85-873a-1f6db6005947","Type":"ContainerDied","Data":"2cb86c0812988580e3ec53056cc3166b868434ae468bbb1b0be38331e2664d39"} Oct 30 21:59:16.248737 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:16.247555 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" event={"ID":"257892d8-a41c-4b4b-96b8-b0985376d220","Type":"ContainerStarted","Data":"69ae64355267197b77e089cf7b03845b5f08f22f2136974aabcdb22ffa794e0c"} Oct 30 21:59:16.248737 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:16.247598 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" event={"ID":"257892d8-a41c-4b4b-96b8-b0985376d220","Type":"ContainerStarted","Data":"b6b9cfb4daecb36beecb1835bc316fe9605dcb574a44ce411d7fc403cc754db8"} Oct 30 21:59:16.475006 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:16.474956 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" podStartSLOduration=2.474940902 podStartE2EDuration="2.474940902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:16.47387523 +0000 UTC m=+1005.155025157" watchObservedRunningTime="2026-10-30 21:59:16.474940902 +0000 UTC m=+1005.156090797" Oct 30 21:59:17.257437 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.257396 4933 generic.go:356] "Generic (PLEG): container finished" podID="257892d8-a41c-4b4b-96b8-b0985376d220" containerID="69ae64355267197b77e089cf7b03845b5f08f22f2136974aabcdb22ffa794e0c" exitCode=0 Oct 30 21:59:17.257616 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.257467 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" event={"ID":"257892d8-a41c-4b4b-96b8-b0985376d220","Type":"ContainerDied","Data":"69ae64355267197b77e089cf7b03845b5f08f22f2136974aabcdb22ffa794e0c"} Oct 30 21:59:17.259715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.259689 4933 generic.go:356] "Generic (PLEG): container finished" podID="eb802b5f-d726-4d17-ad7c-a26a76e9590f" containerID="1467edac65c36e199244b07748cbc99337c375d1d4f37e09679d394a554d9528" exitCode=0 Oct 30 21:59:17.259836 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.259767 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" event={"ID":"eb802b5f-d726-4d17-ad7c-a26a76e9590f","Type":"ContainerDied","Data":"1467edac65c36e199244b07748cbc99337c375d1d4f37e09679d394a554d9528"} Oct 30 21:59:17.424901 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.424878 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:17.568351 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.568317 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9832f67b-f68e-4d85-873a-1f6db6005947-kube-api-access-thp6z\" (UniqueName: \"kubernetes.io/projected/9832f67b-f68e-4d85-873a-1f6db6005947-kube-api-access-thp6z\") pod \"9832f67b-f68e-4d85-873a-1f6db6005947\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " Oct 30 21:59:17.568841 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.568382 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-openshift-ca\") pod \"9832f67b-f68e-4d85-873a-1f6db6005947\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " Oct 30 21:59:17.568841 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.568485 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-xks-ca\") pod \"9832f67b-f68e-4d85-873a-1f6db6005947\" (UID: \"9832f67b-f68e-4d85-873a-1f6db6005947\") " Oct 30 21:59:17.568946 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.568833 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-openshift-ca" pod "9832f67b-f68e-4d85-873a-1f6db6005947" (UID: "9832f67b-f68e-4d85-873a-1f6db6005947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:17.568997 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.568982 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.570972 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.570910 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-xks-ca" pod "9832f67b-f68e-4d85-873a-1f6db6005947" (UID: "9832f67b-f68e-4d85-873a-1f6db6005947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:17.571090 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.571050 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9832f67b-f68e-4d85-873a-1f6db6005947-kube-api-access-thp6z" pod "9832f67b-f68e-4d85-873a-1f6db6005947" (UID: "9832f67b-f68e-4d85-873a-1f6db6005947"). InnerVolumeSpecName "kube-api-access-thp6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:17.672941 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.672905 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thp6z\" (UniqueName: \"kubernetes.io/projected/9832f67b-f68e-4d85-873a-1f6db6005947-kube-api-access-thp6z\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.673152 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.673134 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9832f67b-f68e-4d85-873a-1f6db6005947-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:17.884757 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.884679 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n"] Oct 30 21:59:17.885197 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.885186 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9832f67b-f68e-4d85-873a-1f6db6005947" containerName="revoke-keys" Oct 30 21:59:17.885245 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.885200 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9832f67b-f68e-4d85-873a-1f6db6005947" containerName="revoke-keys" Oct 30 21:59:17.885309 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.885299 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9832f67b-f68e-4d85-873a-1f6db6005947" containerName="revoke-keys" Oct 30 21:59:17.888844 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.888827 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:17.984548 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:17.984478 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n"] Oct 30 21:59:18.077754 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.077719 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81900d6-5680-46cc-8815-387c4fa73013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.077912 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.077847 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.077912 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.077890 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.077912 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.077909 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.078009 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.077931 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.078009 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.077946 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmk7\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-kube-api-access-8rmk7\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.078109 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.078089 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81900d6-5680-46cc-8815-387c4fa73013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.078207 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.078191 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.078238 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.078222 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179476 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179392 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179476 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179430 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179502 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81900d6-5680-46cc-8815-387c4fa73013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179546 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179586 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179605 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179633 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179657 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmk7\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-kube-api-access-8rmk7\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.179735 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179698 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81900d6-5680-46cc-8815-387c4fa73013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.180107 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.179992 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.180404 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.180360 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.180532 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.180422 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.180532 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.180429 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81900d6-5680-46cc-8815-387c4fa73013-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.180675 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.180650 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.181953 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.181930 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.182281 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.182260 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81900d6-5680-46cc-8815-387c4fa73013-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.237503 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.237456 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.237715 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.237694 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmk7\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-kube-api-access-8rmk7\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.268688 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.268666 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" Oct 30 21:59:18.268800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.268686 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r" event={"ID":"9832f67b-f68e-4d85-873a-1f6db6005947","Type":"ContainerDied","Data":"246c3dc141b06b69d4b494c605c1cbfa04d0e8fd20c64f5226f04c8d27948b97"} Oct 30 21:59:18.268800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.268741 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246c3dc141b06b69d4b494c605c1cbfa04d0e8fd20c64f5226f04c8d27948b97" Oct 30 21:59:18.445425 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.445402 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:18.462997 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.462928 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:18.502703 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.502674 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:18.583055 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.583025 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-xks-ca\") pod \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " Oct 30 21:59:18.583463 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.583150 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257892d8-a41c-4b4b-96b8-b0985376d220-kube-api-access-zx77w\" (UniqueName: \"kubernetes.io/projected/257892d8-a41c-4b4b-96b8-b0985376d220-kube-api-access-zx77w\") pod \"257892d8-a41c-4b4b-96b8-b0985376d220\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " Oct 30 21:59:18.583463 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.583186 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-openshift-ca\") pod \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " Oct 30 21:59:18.583463 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.583250 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb802b5f-d726-4d17-ad7c-a26a76e9590f-kube-api-access-wshsc\" (UniqueName: \"kubernetes.io/projected/eb802b5f-d726-4d17-ad7c-a26a76e9590f-kube-api-access-wshsc\") pod \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\" (UID: \"eb802b5f-d726-4d17-ad7c-a26a76e9590f\") " Oct 30 21:59:18.583463 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.583300 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-openshift-ca\") pod \"257892d8-a41c-4b4b-96b8-b0985376d220\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " Oct 30 21:59:18.583463 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.583370 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-xks-ca\") pod \"257892d8-a41c-4b4b-96b8-b0985376d220\" (UID: \"257892d8-a41c-4b4b-96b8-b0985376d220\") " Oct 30 21:59:18.584207 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.583778 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-openshift-ca" pod "eb802b5f-d726-4d17-ad7c-a26a76e9590f" (UID: "eb802b5f-d726-4d17-ad7c-a26a76e9590f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:18.584479 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.584452 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-openshift-ca" pod "257892d8-a41c-4b4b-96b8-b0985376d220" (UID: "257892d8-a41c-4b4b-96b8-b0985376d220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:18.586054 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.586027 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257892d8-a41c-4b4b-96b8-b0985376d220-kube-api-access-zx77w" pod "257892d8-a41c-4b4b-96b8-b0985376d220" (UID: "257892d8-a41c-4b4b-96b8-b0985376d220"). InnerVolumeSpecName "kube-api-access-zx77w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:18.586817 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.586793 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb802b5f-d726-4d17-ad7c-a26a76e9590f-kube-api-access-wshsc" pod "eb802b5f-d726-4d17-ad7c-a26a76e9590f" (UID: "eb802b5f-d726-4d17-ad7c-a26a76e9590f"). InnerVolumeSpecName "kube-api-access-wshsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:18.587743 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.587719 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-xks-ca" pod "257892d8-a41c-4b4b-96b8-b0985376d220" (UID: "257892d8-a41c-4b4b-96b8-b0985376d220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:18.587850 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.587828 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-xks-ca" pod "eb802b5f-d726-4d17-ad7c-a26a76e9590f" (UID: "eb802b5f-d726-4d17-ad7c-a26a76e9590f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:18.685608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.685518 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.685608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.685547 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.685608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.685557 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx77w\" (UniqueName: \"kubernetes.io/projected/257892d8-a41c-4b4b-96b8-b0985376d220-kube-api-access-zx77w\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.685608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.685567 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb802b5f-d726-4d17-ad7c-a26a76e9590f-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.685608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.685576 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wshsc\" (UniqueName: \"kubernetes.io/projected/eb802b5f-d726-4d17-ad7c-a26a76e9590f-kube-api-access-wshsc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.685608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.685590 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257892d8-a41c-4b4b-96b8-b0985376d220-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:18.932848 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:18.932818 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81900d6_5680_46cc_8815_387c4fa73013.slice/crio-bb575b6c9b1a39f42ac717a3db4b988ce6bb9b53ffd04bbabde5885d957aca9f WatchSource:0}: Error finding container bb575b6c9b1a39f42ac717a3db4b988ce6bb9b53ffd04bbabde5885d957aca9f: Status 404 returned error can't find the container with id bb575b6c9b1a39f42ac717a3db4b988ce6bb9b53ffd04bbabde5885d957aca9f Oct 30 21:59:18.933253 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.933232 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n"] Oct 30 21:59:18.935280 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.935252 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:18.935360 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.935312 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:18.935360 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:18.935341 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:19.282553 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.282517 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" event={"ID":"c81900d6-5680-46cc-8815-387c4fa73013","Type":"ContainerStarted","Data":"55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49"} Oct 30 21:59:19.282749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.282584 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" event={"ID":"c81900d6-5680-46cc-8815-387c4fa73013","Type":"ContainerStarted","Data":"bb575b6c9b1a39f42ac717a3db4b988ce6bb9b53ffd04bbabde5885d957aca9f"} Oct 30 21:59:19.284092 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.284069 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" Oct 30 21:59:19.284225 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.284103 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t" event={"ID":"257892d8-a41c-4b4b-96b8-b0985376d220","Type":"ContainerDied","Data":"b6b9cfb4daecb36beecb1835bc316fe9605dcb574a44ce411d7fc403cc754db8"} Oct 30 21:59:19.284225 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.284147 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6b9cfb4daecb36beecb1835bc316fe9605dcb574a44ce411d7fc403cc754db8" Oct 30 21:59:19.285654 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.285628 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" Oct 30 21:59:19.285654 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.285629 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl" event={"ID":"eb802b5f-d726-4d17-ad7c-a26a76e9590f","Type":"ContainerDied","Data":"e855c15a8ebfafd3a865f2623ccd915c5a5a048eaec78f1e25cf871f0d0163e6"} Oct 30 21:59:19.285831 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.285671 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e855c15a8ebfafd3a865f2623ccd915c5a5a048eaec78f1e25cf871f0d0163e6" Oct 30 21:59:19.403158 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.403111 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podStartSLOduration=2.403095943 podStartE2EDuration="2.403095943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:19.400536155 +0000 UTC m=+1008.081686051" watchObservedRunningTime="2026-10-30 21:59:19.403095943 +0000 UTC m=+1008.084245837" Oct 30 21:59:19.502929 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.502840 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:19.504664 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.504640 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:19.504800 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:19.504677 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:20.503704 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:20.503671 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:20.504230 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:20.503727 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:21.503403 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:21.503367 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:21.503597 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:21.503421 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:22.503684 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:22.503648 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:22.504073 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:22.503733 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:23.503860 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:23.503833 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:23.504288 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:23.503886 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:24.503164 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:24.503132 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:24.503341 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:24.503197 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:25.504012 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:25.503975 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:25.504424 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:25.504038 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:26.503568 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:26.503537 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 21:59:26.503753 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:26.503614 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 21:59:27.508189 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:27.508157 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:27.508688 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:27.508600 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:27.509176 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:27.509158 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 21:59:28.543626 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:59:28.543589 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 21:59:30.704750 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.704719 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx"] Oct 30 21:59:30.705197 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.705185 4933 cpu_manager.go:414] "containerMap: removing container" podUID="eb802b5f-d726-4d17-ad7c-a26a76e9590f" containerName="revoke-keys" Oct 30 21:59:30.705197 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.705197 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb802b5f-d726-4d17-ad7c-a26a76e9590f" containerName="revoke-keys" Oct 30 21:59:30.705273 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.705209 4933 cpu_manager.go:414] "containerMap: removing container" podUID="257892d8-a41c-4b4b-96b8-b0985376d220" containerName="revoke-keys" Oct 30 21:59:30.705273 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.705214 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257892d8-a41c-4b4b-96b8-b0985376d220" containerName="revoke-keys" Oct 30 21:59:30.705339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.705294 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb802b5f-d726-4d17-ad7c-a26a76e9590f" containerName="revoke-keys" Oct 30 21:59:30.705339 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.705303 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="257892d8-a41c-4b4b-96b8-b0985376d220" containerName="revoke-keys" Oct 30 21:59:30.709996 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.709980 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.735869 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.735834 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.736016 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.735901 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpb5\" (UniqueName: \"kubernetes.io/projected/5a14f46a-5ee8-484f-891e-ad7433c0196c-kube-api-access-2zpb5\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.736082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.736036 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.767207 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.767178 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx"] Oct 30 21:59:30.812573 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.812546 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:30.812730 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.812650 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:30.812774 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.812739 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:30.818094 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.818076 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:59:30.836817 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.836776 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.836959 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.836836 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpb5\" (UniqueName: \"kubernetes.io/projected/5a14f46a-5ee8-484f-891e-ad7433c0196c-kube-api-access-2zpb5\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.836959 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.836906 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.837816 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.837792 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.840185 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.840161 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:30.895149 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:30.895124 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpb5\" (UniqueName: \"kubernetes.io/projected/5a14f46a-5ee8-484f-891e-ad7433c0196c-kube-api-access-2zpb5\") pod \"maas-api-revoke-sub-39d9e865-r88fx\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:31.023455 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.023379 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:31.412751 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:31.412718 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a14f46a_5ee8_484f_891e_ad7433c0196c.slice/crio-6665d5476b4e3d8250851076b285a539d57f931315ba41d56e3981227e2ca7e3 WatchSource:0}: Error finding container 6665d5476b4e3d8250851076b285a539d57f931315ba41d56e3981227e2ca7e3: Status 404 returned error can't find the container with id 6665d5476b4e3d8250851076b285a539d57f931315ba41d56e3981227e2ca7e3 Oct 30 21:59:31.413137 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.413121 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx"] Oct 30 21:59:31.445220 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.445195 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp"] Oct 30 21:59:31.451046 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.451025 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.524135 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.524107 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp"] Oct 30 21:59:31.544780 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.544751 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.544928 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.544827 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.545244 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.545220 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8ln\" (UniqueName: \"kubernetes.io/projected/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-kube-api-access-7g8ln\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.646376 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.646347 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.646558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.646417 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.646558 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.646442 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8ln\" (UniqueName: \"kubernetes.io/projected/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-kube-api-access-7g8ln\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.647006 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.646985 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.649109 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.649082 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.705301 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.705195 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8ln\" (UniqueName: \"kubernetes.io/projected/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-kube-api-access-7g8ln\") pod \"maas-api-revoke-sub-ae607328-z62wp\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:31.777633 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:31.777608 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:32.158327 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:32.158298 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefba10c2_e5d7_4ad0_9fe5_a7bf36639284.slice/crio-2d38ea07dbbdab2087d0163c39a65de02aaab2efacebc69b7306e5b6936d13d0 WatchSource:0}: Error finding container 2d38ea07dbbdab2087d0163c39a65de02aaab2efacebc69b7306e5b6936d13d0: Status 404 returned error can't find the container with id 2d38ea07dbbdab2087d0163c39a65de02aaab2efacebc69b7306e5b6936d13d0 Oct 30 21:59:32.160520 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.160465 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp"] Oct 30 21:59:32.408980 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.408901 4933 generic.go:356] "Generic (PLEG): container finished" podID="5a14f46a-5ee8-484f-891e-ad7433c0196c" containerID="1786b6d67ff49b3372432e42ce0dcc6dbcfdfce49cf9ecb4bcf6b1a1d3133c7c" exitCode=0 Oct 30 21:59:32.408980 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.408968 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" event={"ID":"5a14f46a-5ee8-484f-891e-ad7433c0196c","Type":"ContainerDied","Data":"1786b6d67ff49b3372432e42ce0dcc6dbcfdfce49cf9ecb4bcf6b1a1d3133c7c"} Oct 30 21:59:32.409210 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.409021 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" event={"ID":"5a14f46a-5ee8-484f-891e-ad7433c0196c","Type":"ContainerStarted","Data":"6665d5476b4e3d8250851076b285a539d57f931315ba41d56e3981227e2ca7e3"} Oct 30 21:59:32.410573 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.410541 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" event={"ID":"efba10c2-e5d7-4ad0-9fe5-a7bf36639284","Type":"ContainerStarted","Data":"2d38ea07dbbdab2087d0163c39a65de02aaab2efacebc69b7306e5b6936d13d0"} Oct 30 21:59:32.759858 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.759769 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk"] Oct 30 21:59:32.764664 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.764646 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.818727 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.818700 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk"] Oct 30 21:59:32.861719 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.861677 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dsb7\" (UniqueName: \"kubernetes.io/projected/da21d38b-237d-43d6-8b28-d2cde25debb1-kube-api-access-2dsb7\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.861940 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.861915 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.862024 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.861960 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.963087 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.963050 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.963242 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.963099 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.963242 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.963163 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dsb7\" (UniqueName: \"kubernetes.io/projected/da21d38b-237d-43d6-8b28-d2cde25debb1-kube-api-access-2dsb7\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.963982 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.963957 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:32.965723 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:32.965701 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:33.020455 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.020402 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dsb7\" (UniqueName: \"kubernetes.io/projected/da21d38b-237d-43d6-8b28-d2cde25debb1-kube-api-access-2dsb7\") pod \"maas-api-revoke-sub-2d3be938-zxzxk\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:33.081434 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.081413 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:33.418217 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.418185 4933 generic.go:356] "Generic (PLEG): container finished" podID="efba10c2-e5d7-4ad0-9fe5-a7bf36639284" containerID="c774f2035afc85d82e8094eaa27ef97e0ac5c945d1dbfbcefabd278e8115043f" exitCode=0 Oct 30 21:59:33.418385 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.418276 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" event={"ID":"efba10c2-e5d7-4ad0-9fe5-a7bf36639284","Type":"ContainerDied","Data":"c774f2035afc85d82e8094eaa27ef97e0ac5c945d1dbfbcefabd278e8115043f"} Oct 30 21:59:33.465920 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:33.465885 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda21d38b_237d_43d6_8b28_d2cde25debb1.slice/crio-b2ce4a7170c65b8868717399f8689f8375c0bbdf14def31cd88f92105f1b35ed WatchSource:0}: Error finding container b2ce4a7170c65b8868717399f8689f8375c0bbdf14def31cd88f92105f1b35ed: Status 404 returned error can't find the container with id b2ce4a7170c65b8868717399f8689f8375c0bbdf14def31cd88f92105f1b35ed Oct 30 21:59:33.470586 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.470556 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk"] Oct 30 21:59:33.583875 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.583845 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:33.669776 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.669699 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-xks-ca\") pod \"5a14f46a-5ee8-484f-891e-ad7433c0196c\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " Oct 30 21:59:33.669908 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.669834 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a14f46a-5ee8-484f-891e-ad7433c0196c-kube-api-access-2zpb5\" (UniqueName: \"kubernetes.io/projected/5a14f46a-5ee8-484f-891e-ad7433c0196c-kube-api-access-2zpb5\") pod \"5a14f46a-5ee8-484f-891e-ad7433c0196c\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " Oct 30 21:59:33.669908 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.669898 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-openshift-ca\") pod \"5a14f46a-5ee8-484f-891e-ad7433c0196c\" (UID: \"5a14f46a-5ee8-484f-891e-ad7433c0196c\") " Oct 30 21:59:33.670284 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.670257 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-openshift-ca" pod "5a14f46a-5ee8-484f-891e-ad7433c0196c" (UID: "5a14f46a-5ee8-484f-891e-ad7433c0196c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:33.672184 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.672161 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a14f46a-5ee8-484f-891e-ad7433c0196c-kube-api-access-2zpb5" pod "5a14f46a-5ee8-484f-891e-ad7433c0196c" (UID: "5a14f46a-5ee8-484f-891e-ad7433c0196c"). InnerVolumeSpecName "kube-api-access-2zpb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:33.672621 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.672588 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-xks-ca" pod "5a14f46a-5ee8-484f-891e-ad7433c0196c" (UID: "5a14f46a-5ee8-484f-891e-ad7433c0196c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:33.771989 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.771955 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.771989 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.771987 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpb5\" (UniqueName: \"kubernetes.io/projected/5a14f46a-5ee8-484f-891e-ad7433c0196c-kube-api-access-2zpb5\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:33.772384 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:33.772002 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a14f46a-5ee8-484f-891e-ad7433c0196c-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:34.426295 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.426259 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" event={"ID":"da21d38b-237d-43d6-8b28-d2cde25debb1","Type":"ContainerStarted","Data":"f08a76d36aed49b73e976f62b46bb0deada9f4e5a2dfe756e941526681a04503"} Oct 30 21:59:34.426475 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.426352 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" event={"ID":"da21d38b-237d-43d6-8b28-d2cde25debb1","Type":"ContainerStarted","Data":"b2ce4a7170c65b8868717399f8689f8375c0bbdf14def31cd88f92105f1b35ed"} Oct 30 21:59:34.427818 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.427799 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" Oct 30 21:59:34.427907 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.427840 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx" event={"ID":"5a14f46a-5ee8-484f-891e-ad7433c0196c","Type":"ContainerDied","Data":"6665d5476b4e3d8250851076b285a539d57f931315ba41d56e3981227e2ca7e3"} Oct 30 21:59:34.427907 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.427870 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6665d5476b4e3d8250851076b285a539d57f931315ba41d56e3981227e2ca7e3" Oct 30 21:59:34.543742 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.543679 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" podStartSLOduration=2.543660368 podStartE2EDuration="2.543660368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:34.542025123 +0000 UTC m=+1023.223175018" watchObservedRunningTime="2026-10-30 21:59:34.543660368 +0000 UTC m=+1023.224810263" Oct 30 21:59:34.591254 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.591222 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:34.681448 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.681378 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-kube-api-access-7g8ln\" (UniqueName: \"kubernetes.io/projected/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-kube-api-access-7g8ln\") pod \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " Oct 30 21:59:34.681448 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.681441 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-openshift-ca\") pod \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " Oct 30 21:59:34.681686 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.681525 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-xks-ca\") pod \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\" (UID: \"efba10c2-e5d7-4ad0-9fe5-a7bf36639284\") " Oct 30 21:59:34.681768 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.681745 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-openshift-ca" pod "efba10c2-e5d7-4ad0-9fe5-a7bf36639284" (UID: "efba10c2-e5d7-4ad0-9fe5-a7bf36639284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:34.681852 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.681834 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:34.683742 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.683718 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-xks-ca" pod "efba10c2-e5d7-4ad0-9fe5-a7bf36639284" (UID: "efba10c2-e5d7-4ad0-9fe5-a7bf36639284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:34.683848 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.683806 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-kube-api-access-7g8ln" pod "efba10c2-e5d7-4ad0-9fe5-a7bf36639284" (UID: "efba10c2-e5d7-4ad0-9fe5-a7bf36639284"). InnerVolumeSpecName "kube-api-access-7g8ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:34.782642 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.782614 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8ln\" (UniqueName: \"kubernetes.io/projected/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-kube-api-access-7g8ln\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:34.782642 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:34.782641 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efba10c2-e5d7-4ad0-9fe5-a7bf36639284-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:35.438408 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:35.438377 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" Oct 30 21:59:35.438645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:35.438372 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp" event={"ID":"efba10c2-e5d7-4ad0-9fe5-a7bf36639284","Type":"ContainerDied","Data":"2d38ea07dbbdab2087d0163c39a65de02aaab2efacebc69b7306e5b6936d13d0"} Oct 30 21:59:35.438645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:35.438569 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d38ea07dbbdab2087d0163c39a65de02aaab2efacebc69b7306e5b6936d13d0" Oct 30 21:59:35.441537 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:35.440441 4933 generic.go:356] "Generic (PLEG): container finished" podID="da21d38b-237d-43d6-8b28-d2cde25debb1" containerID="f08a76d36aed49b73e976f62b46bb0deada9f4e5a2dfe756e941526681a04503" exitCode=0 Oct 30 21:59:35.441537 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:35.440530 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" event={"ID":"da21d38b-237d-43d6-8b28-d2cde25debb1","Type":"ContainerDied","Data":"f08a76d36aed49b73e976f62b46bb0deada9f4e5a2dfe756e941526681a04503"} Oct 30 21:59:36.619669 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.619635 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:36.704353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.704283 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da21d38b-237d-43d6-8b28-d2cde25debb1-kube-api-access-2dsb7\" (UniqueName: \"kubernetes.io/projected/da21d38b-237d-43d6-8b28-d2cde25debb1-kube-api-access-2dsb7\") pod \"da21d38b-237d-43d6-8b28-d2cde25debb1\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " Oct 30 21:59:36.704353 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.704328 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-openshift-ca\") pod \"da21d38b-237d-43d6-8b28-d2cde25debb1\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " Oct 30 21:59:36.704588 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.704379 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-xks-ca\") pod \"da21d38b-237d-43d6-8b28-d2cde25debb1\" (UID: \"da21d38b-237d-43d6-8b28-d2cde25debb1\") " Oct 30 21:59:36.704690 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.704668 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-openshift-ca" pod "da21d38b-237d-43d6-8b28-d2cde25debb1" (UID: "da21d38b-237d-43d6-8b28-d2cde25debb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:36.704878 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.704860 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:36.706612 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.706590 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da21d38b-237d-43d6-8b28-d2cde25debb1-kube-api-access-2dsb7" pod "da21d38b-237d-43d6-8b28-d2cde25debb1" (UID: "da21d38b-237d-43d6-8b28-d2cde25debb1"). InnerVolumeSpecName "kube-api-access-2dsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:36.706828 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.706813 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-xks-ca" pod "da21d38b-237d-43d6-8b28-d2cde25debb1" (UID: "da21d38b-237d-43d6-8b28-d2cde25debb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:36.805685 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.805655 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dsb7\" (UniqueName: \"kubernetes.io/projected/da21d38b-237d-43d6-8b28-d2cde25debb1-kube-api-access-2dsb7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:36.805685 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:36.805680 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da21d38b-237d-43d6-8b28-d2cde25debb1-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:37.456536 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:37.456503 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" Oct 30 21:59:37.456728 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:37.456509 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk" event={"ID":"da21d38b-237d-43d6-8b28-d2cde25debb1","Type":"ContainerDied","Data":"b2ce4a7170c65b8868717399f8689f8375c0bbdf14def31cd88f92105f1b35ed"} Oct 30 21:59:37.456728 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:37.456618 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ce4a7170c65b8868717399f8689f8375c0bbdf14def31cd88f92105f1b35ed" Oct 30 21:59:38.208130 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208096 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww"] Oct 30 21:59:38.208613 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208599 4933 cpu_manager.go:414] "containerMap: removing container" podUID="efba10c2-e5d7-4ad0-9fe5-a7bf36639284" containerName="revoke-keys" Oct 30 21:59:38.208676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208614 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efba10c2-e5d7-4ad0-9fe5-a7bf36639284" containerName="revoke-keys" Oct 30 21:59:38.208676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208631 4933 cpu_manager.go:414] "containerMap: removing container" podUID="5a14f46a-5ee8-484f-891e-ad7433c0196c" containerName="revoke-keys" Oct 30 21:59:38.208676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208636 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a14f46a-5ee8-484f-891e-ad7433c0196c" containerName="revoke-keys" Oct 30 21:59:38.208676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208653 4933 cpu_manager.go:414] "containerMap: removing container" podUID="da21d38b-237d-43d6-8b28-d2cde25debb1" containerName="revoke-keys" Oct 30 21:59:38.208676 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208658 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da21d38b-237d-43d6-8b28-d2cde25debb1" containerName="revoke-keys" Oct 30 21:59:38.208829 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208746 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="da21d38b-237d-43d6-8b28-d2cde25debb1" containerName="revoke-keys" Oct 30 21:59:38.208829 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208758 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="efba10c2-e5d7-4ad0-9fe5-a7bf36639284" containerName="revoke-keys" Oct 30 21:59:38.208829 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.208765 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a14f46a-5ee8-484f-891e-ad7433c0196c" containerName="revoke-keys" Oct 30 21:59:38.212304 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.212288 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.273575 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.273542 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww"] Oct 30 21:59:38.322081 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322049 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7efe19-eacd-4488-81c6-44d87f02ac64-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322081 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322080 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322273 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322148 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322273 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322231 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322340 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322303 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322378 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322353 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322411 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322384 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9nk7\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-kube-api-access-c9nk7\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322447 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322415 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.322480 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.322465 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423321 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423290 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9nk7\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-kube-api-access-c9nk7\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423467 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423329 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423467 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423345 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423467 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423397 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7efe19-eacd-4488-81c6-44d87f02ac64-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423467 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423414 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423473 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423536 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423600 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423749 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423654 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423960 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423783 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.423960 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423834 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.424042 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.423962 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.424082 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.424060 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.424274 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.424250 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7efe19-eacd-4488-81c6-44d87f02ac64-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.426144 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.426120 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.426236 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.426160 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.481408 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.481325 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.481645 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.481628 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9nk7\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-kube-api-access-c9nk7\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.532321 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.532295 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p"] Oct 30 21:59:38.533312 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.533291 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:38.537865 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.537724 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.609101 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.606399 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p"] Oct 30 21:59:38.611076 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.610643 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d516c4-openshift-default-dockercfg-mtv65\"" Oct 30 21:59:38.625918 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.625892 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626042 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.625937 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f015183f-507c-4318-af33-2190e83755e8-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626042 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.626025 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626133 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.626073 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhv7q\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-kube-api-access-dhv7q\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626176 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.626144 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626224 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.626191 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626313 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.626297 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626372 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.626326 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.626436 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.626420 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f015183f-507c-4318-af33-2190e83755e8-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.727719 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.727691 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f015183f-507c-4318-af33-2190e83755e8-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.727891 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.727742 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.727891 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.727771 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f015183f-507c-4318-af33-2190e83755e8-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.727891 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.727795 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.727891 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.727835 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhv7q\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-kube-api-access-dhv7q\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728099 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.727962 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728099 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728047 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728201 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728117 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728201 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728143 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728314 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728225 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728467 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728441 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728608 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728518 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728661 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728632 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.728842 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.728768 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f015183f-507c-4318-af33-2190e83755e8-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.730809 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.730781 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.730927 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.730865 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f015183f-507c-4318-af33-2190e83755e8-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.785993 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.785969 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhv7q\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-kube-api-access-dhv7q\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.786154 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.786136 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.872662 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.872633 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:38.964959 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:38.964931 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7efe19_eacd_4488_81c6_44d87f02ac64.slice/crio-ea228fa2807422117174ee67403002596c483fb7ce4475599b87477f20bfbb2a WatchSource:0}: Error finding container ea228fa2807422117174ee67403002596c483fb7ce4475599b87477f20bfbb2a: Status 404 returned error can't find the container with id ea228fa2807422117174ee67403002596c483fb7ce4475599b87477f20bfbb2a Oct 30 21:59:38.966855 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.966829 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww"] Oct 30 21:59:38.968046 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.968012 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.968134 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.968090 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:38.968209 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:38.968136 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:39.252860 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:39.252826 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf015183f_507c_4318_af33_2190e83755e8.slice/crio-945ae79ca3aa3660b0426e9b3fa1f7e5e5c10d2f8ba196620e612cc62d9da00b WatchSource:0}: Error finding container 945ae79ca3aa3660b0426e9b3fa1f7e5e5c10d2f8ba196620e612cc62d9da00b: Status 404 returned error can't find the container with id 945ae79ca3aa3660b0426e9b3fa1f7e5e5c10d2f8ba196620e612cc62d9da00b Oct 30 21:59:39.253288 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.252962 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p"] Oct 30 21:59:39.256057 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.256017 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:39.256170 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.256105 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:39.256170 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.256148 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 21:59:39.473008 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.472961 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" event={"ID":"f015183f-507c-4318-af33-2190e83755e8","Type":"ContainerStarted","Data":"c4c9d425d65eb676a55e977d1e1de52993d9449a59e5f9506108f7b96a6e72ef"} Oct 30 21:59:39.473168 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.473026 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" event={"ID":"f015183f-507c-4318-af33-2190e83755e8","Type":"ContainerStarted","Data":"945ae79ca3aa3660b0426e9b3fa1f7e5e5c10d2f8ba196620e612cc62d9da00b"} Oct 30 21:59:39.475336 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.475301 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" event={"ID":"0c7efe19-eacd-4488-81c6-44d87f02ac64","Type":"ContainerStarted","Data":"523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614"} Oct 30 21:59:39.475515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.475348 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" event={"ID":"0c7efe19-eacd-4488-81c6-44d87f02ac64","Type":"ContainerStarted","Data":"ea228fa2807422117174ee67403002596c483fb7ce4475599b87477f20bfbb2a"} Oct 30 21:59:39.533945 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.533920 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:39.593551 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.593259 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podStartSLOduration=1.5932406239999999 podStartE2EDuration="1.593240624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:39.591072848 +0000 UTC m=+1028.272222743" watchObservedRunningTime="2026-10-30 21:59:39.593240624 +0000 UTC m=+1028.274390519" Oct 30 21:59:39.711252 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.711202 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podStartSLOduration=1.711188606 podStartE2EDuration="1.711188606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:39.709044933 +0000 UTC m=+1028.390194818" watchObservedRunningTime="2026-10-30 21:59:39.711188606 +0000 UTC m=+1028.392338910" Oct 30 21:59:39.873032 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:39.872936 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:40.539537 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:40.534969 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:59:40.540148 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:40.540111 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:59:40.876578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:40.874987 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:59:40.876578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:40.875065 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:59:41.547520 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:41.542812 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:59:41.547520 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:41.542867 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:59:41.885578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:41.885210 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:59:41.885578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:41.885270 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:59:42.539830 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:42.539796 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:59:42.540118 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:42.540093 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:59:42.873964 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:42.873869 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:59:42.874578 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:42.874542 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:59:43.550518 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:43.549823 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:59:43.550518 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:43.549882 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:59:43.884574 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:43.876860 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:59:43.884574 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:43.876922 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:59:44.557515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:44.555824 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:59:44.557515 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:44.555891 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:59:44.876570 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:44.875834 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:59:44.876570 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:44.875892 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:59:45.538682 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:45.536656 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 21:59:45.538682 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:45.536725 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 21:59:45.873697 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:45.873613 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:59:45.873697 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:45.873677 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:59:46.539897 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.539863 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:46.540613 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.540589 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:46.541389 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.541369 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 21:59:46.653027 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.652989 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974"] Oct 30 21:59:46.656882 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.656867 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.683788 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.683760 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974"] Oct 30 21:59:46.709758 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.709727 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:59:46.709935 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.709920 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:59:46.709975 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.709969 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 21:59:46.710013 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.709995 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:59:46.758813 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.758786 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.758926 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.758821 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bknkx\" (UniqueName: \"kubernetes.io/projected/88e6546e-fc61-47ed-abb7-613e4e2e79ab-kube-api-access-bknkx\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.758926 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.758846 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.860319 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.860249 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.860319 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.860289 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bknkx\" (UniqueName: \"kubernetes.io/projected/88e6546e-fc61-47ed-abb7-613e4e2e79ab-kube-api-access-bknkx\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.860536 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.860323 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.860996 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.860976 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.862752 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.862730 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.877986 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.877962 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:46.878348 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.878322 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:46.878947 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.878929 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 21:59:46.917869 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.917846 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bknkx\" (UniqueName: \"kubernetes.io/projected/88e6546e-fc61-47ed-abb7-613e4e2e79ab-kube-api-access-bknkx\") pod \"maas-api-revoke-sub-76f898c4-9n974\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:46.971066 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:46.971040 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:47.362673 ip-10-0-131-250 kubenswrapper[4933]: W1030 21:59:47.362640 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e6546e_fc61_47ed_abb7_613e4e2e79ab.slice/crio-461779cd7165fa058d3222e1a025cc877e639114981d92f7028eb1be70a67547 WatchSource:0}: Error finding container 461779cd7165fa058d3222e1a025cc877e639114981d92f7028eb1be70a67547: Status 404 returned error can't find the container with id 461779cd7165fa058d3222e1a025cc877e639114981d92f7028eb1be70a67547 Oct 30 21:59:47.363233 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:47.363208 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974"] Oct 30 21:59:47.562045 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:47.562009 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" event={"ID":"88e6546e-fc61-47ed-abb7-613e4e2e79ab","Type":"ContainerStarted","Data":"461779cd7165fa058d3222e1a025cc877e639114981d92f7028eb1be70a67547"} Oct 30 21:59:48.570695 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:48.570661 4933 generic.go:356] "Generic (PLEG): container finished" podID="88e6546e-fc61-47ed-abb7-613e4e2e79ab" containerID="13683740bf0ae5395024b8e873deec52fc548884da23223bb6f339cf99141394" exitCode=0 Oct 30 21:59:48.571100 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:48.570747 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" event={"ID":"88e6546e-fc61-47ed-abb7-613e4e2e79ab","Type":"ContainerDied","Data":"13683740bf0ae5395024b8e873deec52fc548884da23223bb6f339cf99141394"} Oct 30 21:59:49.744381 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.744351 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:49.892989 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.892913 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-xks-ca\") pod \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " Oct 30 21:59:49.893234 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.893212 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e6546e-fc61-47ed-abb7-613e4e2e79ab-kube-api-access-bknkx\" (UniqueName: \"kubernetes.io/projected/88e6546e-fc61-47ed-abb7-613e4e2e79ab-kube-api-access-bknkx\") pod \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " Oct 30 21:59:49.893786 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.893762 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-openshift-ca\") pod \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\" (UID: \"88e6546e-fc61-47ed-abb7-613e4e2e79ab\") " Oct 30 21:59:49.894238 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.894195 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-openshift-ca" pod "88e6546e-fc61-47ed-abb7-613e4e2e79ab" (UID: "88e6546e-fc61-47ed-abb7-613e4e2e79ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:49.897692 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.897664 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-xks-ca" pod "88e6546e-fc61-47ed-abb7-613e4e2e79ab" (UID: "88e6546e-fc61-47ed-abb7-613e4e2e79ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:49.898850 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.898830 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e6546e-fc61-47ed-abb7-613e4e2e79ab-kube-api-access-bknkx" pod "88e6546e-fc61-47ed-abb7-613e4e2e79ab" (UID: "88e6546e-fc61-47ed-abb7-613e4e2e79ab"). InnerVolumeSpecName "kube-api-access-bknkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:49.995262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.995229 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:49.995262 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.995264 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bknkx\" (UniqueName: \"kubernetes.io/projected/88e6546e-fc61-47ed-abb7-613e4e2e79ab-kube-api-access-bknkx\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:49.995474 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:49.995280 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e6546e-fc61-47ed-abb7-613e4e2e79ab-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 21:59:50.585982 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:50.585948 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" Oct 30 21:59:50.586169 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:50.585973 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974" event={"ID":"88e6546e-fc61-47ed-abb7-613e4e2e79ab","Type":"ContainerDied","Data":"461779cd7165fa058d3222e1a025cc877e639114981d92f7028eb1be70a67547"} Oct 30 21:59:50.586169 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:50.586029 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="461779cd7165fa058d3222e1a025cc877e639114981d92f7028eb1be70a67547" Oct 30 21:59:53.648414 ip-10-0-131-250 kubenswrapper[4933]: I1030 21:59:53.648381 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 21:59:58.544547 ip-10-0-131-250 kubenswrapper[4933]: E1030 21:59:58.544473 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:00:00.177470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.177438 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp"] Oct 30 22:00:00.177993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.177978 4933 cpu_manager.go:414] "containerMap: removing container" podUID="88e6546e-fc61-47ed-abb7-613e4e2e79ab" containerName="revoke-keys" Oct 30 22:00:00.178037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.177995 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e6546e-fc61-47ed-abb7-613e4e2e79ab" containerName="revoke-keys" Oct 30 22:00:00.178095 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.178086 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e6546e-fc61-47ed-abb7-613e4e2e79ab" containerName="revoke-keys" Oct 30 22:00:00.182816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.182801 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r"] Oct 30 22:00:00.182932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.182914 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.186805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.186787 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq"] Oct 30 22:00:00.186931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.186906 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.190648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.190537 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.227471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.227446 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx"] Oct 30 22:00:00.231316 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.231301 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7"] Oct 30 22:00:00.231422 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.231405 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.234938 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.234923 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.239238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.239219 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:00:00.239326 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.239273 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:00:00.274563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.274535 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:00:00.279726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.279707 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r"] Oct 30 22:00:00.293722 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.293697 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx"] Oct 30 22:00:00.298470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.298447 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp"] Oct 30 22:00:00.306231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.306206 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdgtd\" (UniqueName: \"kubernetes.io/projected/9862cec8-1f8e-4400-924a-428883f932fc-kube-api-access-fdgtd\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.306347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.306316 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8c2p\" (UniqueName: \"kubernetes.io/projected/b9a2aa35-79f7-4d31-8345-8082258e0aa0-kube-api-access-q8c2p\") pod \"maas-api-key-cleanup-29889960-wrsqq\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.306428 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.306357 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4338b3b-be7d-4f8a-896a-5245bb9deec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.306514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.306435 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dzl\" (UniqueName: \"kubernetes.io/projected/e4338b3b-be7d-4f8a-896a-5245bb9deec4-kube-api-access-75dzl\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.306514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.306463 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a2aa35-79f7-4d31-8345-8082258e0aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-wrsqq\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.306514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.306482 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9862cec8-1f8e-4400-924a-428883f932fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.333601 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.331955 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7"] Oct 30 22:00:00.348756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.348737 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq"] Oct 30 22:00:00.408003 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.407979 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4338b3b-be7d-4f8a-896a-5245bb9deec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.408132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408038 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dzl\" (UniqueName: \"kubernetes.io/projected/e4338b3b-be7d-4f8a-896a-5245bb9deec4-kube-api-access-75dzl\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.408132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408077 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a2aa35-79f7-4d31-8345-8082258e0aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-wrsqq\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.408132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408101 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9862cec8-1f8e-4400-924a-428883f932fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.408132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408121 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.408345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408197 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mchfm\" (UniqueName: \"kubernetes.io/projected/b1554472-bf3c-4941-a468-6fb08f27b445-kube-api-access-mchfm\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.408345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408220 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdn9c\" (UniqueName: \"kubernetes.io/projected/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-kube-api-access-fdn9c\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.408345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408244 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdgtd\" (UniqueName: \"kubernetes.io/projected/9862cec8-1f8e-4400-924a-428883f932fc-kube-api-access-fdgtd\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.408345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408298 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1554472-bf3c-4941-a468-6fb08f27b445-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.408345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408338 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8c2p\" (UniqueName: \"kubernetes.io/projected/b9a2aa35-79f7-4d31-8345-8082258e0aa0-kube-api-access-q8c2p\") pod \"maas-api-key-cleanup-29889960-wrsqq\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.408849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408827 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a2aa35-79f7-4d31-8345-8082258e0aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-wrsqq\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.408909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408828 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4338b3b-be7d-4f8a-896a-5245bb9deec4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.408909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.408884 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9862cec8-1f8e-4400-924a-428883f932fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.466701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.466646 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdgtd\" (UniqueName: \"kubernetes.io/projected/9862cec8-1f8e-4400-924a-428883f932fc-kube-api-access-fdgtd\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.466701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.466683 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dzl\" (UniqueName: \"kubernetes.io/projected/e4338b3b-be7d-4f8a-896a-5245bb9deec4-kube-api-access-75dzl\") pod \"maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.466833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.466745 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8c2p\" (UniqueName: \"kubernetes.io/projected/b9a2aa35-79f7-4d31-8345-8082258e0aa0-kube-api-access-q8c2p\") pod \"maas-api-key-cleanup-29889960-wrsqq\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.497809 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.497790 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:00.506006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.505990 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:00.508833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.508813 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mchfm\" (UniqueName: \"kubernetes.io/projected/b1554472-bf3c-4941-a468-6fb08f27b445-kube-api-access-mchfm\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.508874 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.508836 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdn9c\" (UniqueName: \"kubernetes.io/projected/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-kube-api-access-fdn9c\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.508915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.508882 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1554472-bf3c-4941-a468-6fb08f27b445-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.509032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.509014 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.509483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.509462 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1554472-bf3c-4941-a468-6fb08f27b445-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.509748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.509727 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.512866 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.512844 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:00.596431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.596407 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdn9c\" (UniqueName: \"kubernetes.io/projected/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-kube-api-access-fdn9c\") pod \"maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.601710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.601688 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mchfm\" (UniqueName: \"kubernetes.io/projected/b1554472-bf3c-4941-a468-6fb08f27b445-kube-api-access-mchfm\") pod \"maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.844721 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.844693 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:00.850738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.850722 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:00.892291 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.892260 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp"] Oct 30 22:00:00.946891 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.946216 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r"] Oct 30 22:00:00.949667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:00.949648 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq"] Oct 30 22:00:00.951709 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:00:00.951667 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a2aa35_79f7_4d31_8345_8082258e0aa0.slice/crio-890daeb7fb56994327b3229b92e080e7f17e32b6bff8c34cd430fa22d8a5cc8c WatchSource:0}: Error finding container 890daeb7fb56994327b3229b92e080e7f17e32b6bff8c34cd430fa22d8a5cc8c: Status 404 returned error can't find the container with id 890daeb7fb56994327b3229b92e080e7f17e32b6bff8c34cd430fa22d8a5cc8c Oct 30 22:00:00.953076 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:00:00.953043 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9862cec8_1f8e_4400_924a_428883f932fc.slice/crio-60b47e7bc5db048d4c35a9d04280029b69569baa4ab5a445e63d6940e52656c7 WatchSource:0}: Error finding container 60b47e7bc5db048d4c35a9d04280029b69569baa4ab5a445e63d6940e52656c7: Status 404 returned error can't find the container with id 60b47e7bc5db048d4c35a9d04280029b69569baa4ab5a445e63d6940e52656c7 Oct 30 22:00:01.234981 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:00:01.234954 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1554472_bf3c_4941_a468_6fb08f27b445.slice/crio-20c8553865d8c886348fc863cebb9092325968c3115e27cd376d3ac5adfb3f7b WatchSource:0}: Error finding container 20c8553865d8c886348fc863cebb9092325968c3115e27cd376d3ac5adfb3f7b: Status 404 returned error can't find the container with id 20c8553865d8c886348fc863cebb9092325968c3115e27cd376d3ac5adfb3f7b Oct 30 22:00:01.235368 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.235144 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx"] Oct 30 22:00:01.244457 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.244434 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7"] Oct 30 22:00:01.254068 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:00:01.254026 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fe3ec0_da54_4ecb_bdb2_6c04a339e6d9.slice/crio-51b454a6897b942c9bc8d4210d1195674f1e248dd329ec378c3f5b54d09c978f WatchSource:0}: Error finding container 51b454a6897b942c9bc8d4210d1195674f1e248dd329ec378c3f5b54d09c978f: Status 404 returned error can't find the container with id 51b454a6897b942c9bc8d4210d1195674f1e248dd329ec378c3f5b54d09c978f Oct 30 22:00:01.678826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.678790 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" event={"ID":"b9a2aa35-79f7-4d31-8345-8082258e0aa0","Type":"ContainerStarted","Data":"47c765413da18284f7d97c1bf4e6614c4fa928ee41d576ff34a60258b6f2936b"} Oct 30 22:00:01.679006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.678855 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" event={"ID":"b9a2aa35-79f7-4d31-8345-8082258e0aa0","Type":"ContainerStarted","Data":"890daeb7fb56994327b3229b92e080e7f17e32b6bff8c34cd430fa22d8a5cc8c"} Oct 30 22:00:01.680858 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.680818 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerStarted","Data":"2d368c8373c08472353505d42ef8ebf6a87cfeab6efed7b672e63c993d72482d"} Oct 30 22:00:01.681014 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.680887 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerStarted","Data":"51b454a6897b942c9bc8d4210d1195674f1e248dd329ec378c3f5b54d09c978f"} Oct 30 22:00:01.682645 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.682623 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerStarted","Data":"65a20c0f6cc3ad6a975aa55dff2cd2af61c1769abb69b28e3d99b72d203861e8"} Oct 30 22:00:01.682747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.682669 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerStarted","Data":"54c6395d26b83d3f31985915341864529d74472b1b2a260d355c7a6c469807bb"} Oct 30 22:00:01.684517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.684473 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerStarted","Data":"ad579e9ae4ffb8193d9c630a3cafa0e2efa83feb1d4b861044fd3c2c50283417"} Oct 30 22:00:01.684627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.684544 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerStarted","Data":"20c8553865d8c886348fc863cebb9092325968c3115e27cd376d3ac5adfb3f7b"} Oct 30 22:00:01.686112 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.686092 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerStarted","Data":"2dbb22e6e515fbc33d7af9dac1da9b5f915f0458526f25ab5aea6a320fe3a9aa"} Oct 30 22:00:01.686244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.686129 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerStarted","Data":"60b47e7bc5db048d4c35a9d04280029b69569baa4ab5a445e63d6940e52656c7"} Oct 30 22:00:01.795129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.795071 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" podStartSLOduration=1.795058857 podStartE2EDuration="1.795058857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.793325538 +0000 UTC m=+1050.474475432" watchObservedRunningTime="2026-10-30 22:00:01.795058857 +0000 UTC m=+1050.476208750" Oct 30 22:00:01.910848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:01.910794 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" podStartSLOduration=1.910779808 podStartE2EDuration="1.910779808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.909207537 +0000 UTC m=+1050.590357456" watchObservedRunningTime="2026-10-30 22:00:01.910779808 +0000 UTC m=+1050.591929695" Oct 30 22:00:02.026319 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.026270 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" podStartSLOduration=2.026256464 podStartE2EDuration="2.026256464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.025137438 +0000 UTC m=+1050.706287330" watchObservedRunningTime="2026-10-30 22:00:02.026256464 +0000 UTC m=+1050.707406356" Oct 30 22:00:02.139596 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.139477 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" podStartSLOduration=2.139457764 podStartE2EDuration="2.139457764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.137961255 +0000 UTC m=+1050.819111147" watchObservedRunningTime="2026-10-30 22:00:02.139457764 +0000 UTC m=+1050.820607657" Oct 30 22:00:02.252415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.252354 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" podStartSLOduration=2.252337721 podStartE2EDuration="2.252337721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.251641882 +0000 UTC m=+1050.932791775" watchObservedRunningTime="2026-10-30 22:00:02.252337721 +0000 UTC m=+1050.933487615" Oct 30 22:00:02.695063 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.695022 4933 generic.go:356] "Generic (PLEG): container finished" podID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerID="2d368c8373c08472353505d42ef8ebf6a87cfeab6efed7b672e63c993d72482d" exitCode=60 Oct 30 22:00:02.695228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.695062 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerDied","Data":"2d368c8373c08472353505d42ef8ebf6a87cfeab6efed7b672e63c993d72482d"} Oct 30 22:00:02.695366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.695342 4933 scope.go:122] "RemoveContainer" containerID="2d368c8373c08472353505d42ef8ebf6a87cfeab6efed7b672e63c993d72482d" Oct 30 22:00:02.697347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.697326 4933 generic.go:356] "Generic (PLEG): container finished" podID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerID="65a20c0f6cc3ad6a975aa55dff2cd2af61c1769abb69b28e3d99b72d203861e8" exitCode=60 Oct 30 22:00:02.697439 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.697411 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerDied","Data":"65a20c0f6cc3ad6a975aa55dff2cd2af61c1769abb69b28e3d99b72d203861e8"} Oct 30 22:00:02.697685 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.697667 4933 scope.go:122] "RemoveContainer" containerID="65a20c0f6cc3ad6a975aa55dff2cd2af61c1769abb69b28e3d99b72d203861e8" Oct 30 22:00:02.702164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.702144 4933 generic.go:356] "Generic (PLEG): container finished" podID="b1554472-bf3c-4941-a468-6fb08f27b445" containerID="ad579e9ae4ffb8193d9c630a3cafa0e2efa83feb1d4b861044fd3c2c50283417" exitCode=60 Oct 30 22:00:02.702251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.702228 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerDied","Data":"ad579e9ae4ffb8193d9c630a3cafa0e2efa83feb1d4b861044fd3c2c50283417"} Oct 30 22:00:02.702415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.702401 4933 scope.go:122] "RemoveContainer" containerID="ad579e9ae4ffb8193d9c630a3cafa0e2efa83feb1d4b861044fd3c2c50283417" Oct 30 22:00:02.703829 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.703811 4933 generic.go:356] "Generic (PLEG): container finished" podID="9862cec8-1f8e-4400-924a-428883f932fc" containerID="2dbb22e6e515fbc33d7af9dac1da9b5f915f0458526f25ab5aea6a320fe3a9aa" exitCode=60 Oct 30 22:00:02.703927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.703867 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerDied","Data":"2dbb22e6e515fbc33d7af9dac1da9b5f915f0458526f25ab5aea6a320fe3a9aa"} Oct 30 22:00:02.704107 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.704094 4933 scope.go:122] "RemoveContainer" containerID="2dbb22e6e515fbc33d7af9dac1da9b5f915f0458526f25ab5aea6a320fe3a9aa" Oct 30 22:00:02.705641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.705618 4933 generic.go:356] "Generic (PLEG): container finished" podID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerID="47c765413da18284f7d97c1bf4e6614c4fa928ee41d576ff34a60258b6f2936b" exitCode=60 Oct 30 22:00:02.705741 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.705715 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" event={"ID":"b9a2aa35-79f7-4d31-8345-8082258e0aa0","Type":"ContainerDied","Data":"47c765413da18284f7d97c1bf4e6614c4fa928ee41d576ff34a60258b6f2936b"} Oct 30 22:00:02.705958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:02.705940 4933 scope.go:122] "RemoveContainer" containerID="47c765413da18284f7d97c1bf4e6614c4fa928ee41d576ff34a60258b6f2936b" Oct 30 22:00:03.714878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.714843 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerStarted","Data":"d8088c9f794b7b1c6db7246ed451d263c53763341d476d61c746bf96527df950"} Oct 30 22:00:03.716823 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.716798 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerStarted","Data":"602e43601d9ea0ed7834fead83d536796fdaed6f9f23684c8218c07a7667b454"} Oct 30 22:00:03.718585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.718563 4933 generic.go:356] "Generic (PLEG): container finished" podID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerID="91fb65cad619a27822e52cc952167067c2b39e8132d0975c8c00a0fbb7f325f3" exitCode=60 Oct 30 22:00:03.718688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.718640 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" event={"ID":"b9a2aa35-79f7-4d31-8345-8082258e0aa0","Type":"ContainerDied","Data":"91fb65cad619a27822e52cc952167067c2b39e8132d0975c8c00a0fbb7f325f3"} Oct 30 22:00:03.718768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.718695 4933 scope.go:122] "RemoveContainer" containerID="47c765413da18284f7d97c1bf4e6614c4fa928ee41d576ff34a60258b6f2936b" Oct 30 22:00:03.718929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.718910 4933 scope.go:122] "RemoveContainer" containerID="91fb65cad619a27822e52cc952167067c2b39e8132d0975c8c00a0fbb7f325f3" Oct 30 22:00:03.719167 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:03.719136 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-wrsqq_odh-ai-gateway-infra(b9a2aa35-79f7-4d31-8345-8082258e0aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" Oct 30 22:00:03.720838 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.720812 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerStarted","Data":"beb4227ea657e70d78418c5d6f65a368e77021bdabd334e4cbd021c10776335a"} Oct 30 22:00:03.722892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:03.722870 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerStarted","Data":"73b40df86b710e22ac0100e911018998557d9da844ad8c899907adfb0fab98c3"} Oct 30 22:00:04.732771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.732738 4933 generic.go:356] "Generic (PLEG): container finished" podID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerID="beb4227ea657e70d78418c5d6f65a368e77021bdabd334e4cbd021c10776335a" exitCode=60 Oct 30 22:00:04.733251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.732825 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerDied","Data":"beb4227ea657e70d78418c5d6f65a368e77021bdabd334e4cbd021c10776335a"} Oct 30 22:00:04.733251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.732889 4933 scope.go:122] "RemoveContainer" containerID="2d368c8373c08472353505d42ef8ebf6a87cfeab6efed7b672e63c993d72482d" Oct 30 22:00:04.733251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.733026 4933 scope.go:122] "RemoveContainer" containerID="beb4227ea657e70d78418c5d6f65a368e77021bdabd334e4cbd021c10776335a" Oct 30 22:00:04.733397 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:04.733283 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7_odh-ai-gateway-infra(57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" Oct 30 22:00:04.735698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.735673 4933 generic.go:356] "Generic (PLEG): container finished" podID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerID="73b40df86b710e22ac0100e911018998557d9da844ad8c899907adfb0fab98c3" exitCode=60 Oct 30 22:00:04.735833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.735771 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerDied","Data":"73b40df86b710e22ac0100e911018998557d9da844ad8c899907adfb0fab98c3"} Oct 30 22:00:04.735959 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.735943 4933 scope.go:122] "RemoveContainer" containerID="73b40df86b710e22ac0100e911018998557d9da844ad8c899907adfb0fab98c3" Oct 30 22:00:04.736270 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:04.736247 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp_odh-ai-gateway-infra(e4338b3b-be7d-4f8a-896a-5245bb9deec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" Oct 30 22:00:04.738033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.738005 4933 generic.go:356] "Generic (PLEG): container finished" podID="b1554472-bf3c-4941-a468-6fb08f27b445" containerID="d8088c9f794b7b1c6db7246ed451d263c53763341d476d61c746bf96527df950" exitCode=60 Oct 30 22:00:04.738151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.738047 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerDied","Data":"d8088c9f794b7b1c6db7246ed451d263c53763341d476d61c746bf96527df950"} Oct 30 22:00:04.738294 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.738275 4933 scope.go:122] "RemoveContainer" containerID="d8088c9f794b7b1c6db7246ed451d263c53763341d476d61c746bf96527df950" Oct 30 22:00:04.738535 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:04.738512 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx_odh-ai-gateway-infra(b1554472-bf3c-4941-a468-6fb08f27b445)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" Oct 30 22:00:04.740273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.740222 4933 generic.go:356] "Generic (PLEG): container finished" podID="9862cec8-1f8e-4400-924a-428883f932fc" containerID="602e43601d9ea0ed7834fead83d536796fdaed6f9f23684c8218c07a7667b454" exitCode=60 Oct 30 22:00:04.740390 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.740300 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerDied","Data":"602e43601d9ea0ed7834fead83d536796fdaed6f9f23684c8218c07a7667b454"} Oct 30 22:00:04.740527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.740509 4933 scope.go:122] "RemoveContainer" containerID="602e43601d9ea0ed7834fead83d536796fdaed6f9f23684c8218c07a7667b454" Oct 30 22:00:04.740773 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:04.740747 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r_odh-ai-gateway-infra(9862cec8-1f8e-4400-924a-428883f932fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" podUID="9862cec8-1f8e-4400-924a-428883f932fc" Oct 30 22:00:04.742462 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.742446 4933 scope.go:122] "RemoveContainer" containerID="91fb65cad619a27822e52cc952167067c2b39e8132d0975c8c00a0fbb7f325f3" Oct 30 22:00:04.742712 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:04.742691 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-wrsqq_odh-ai-gateway-infra(b9a2aa35-79f7-4d31-8345-8082258e0aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" Oct 30 22:00:04.747030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.747011 4933 scope.go:122] "RemoveContainer" containerID="65a20c0f6cc3ad6a975aa55dff2cd2af61c1769abb69b28e3d99b72d203861e8" Oct 30 22:00:04.762389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.762370 4933 scope.go:122] "RemoveContainer" containerID="ad579e9ae4ffb8193d9c630a3cafa0e2efa83feb1d4b861044fd3c2c50283417" Oct 30 22:00:04.775626 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:04.775606 4933 scope.go:122] "RemoveContainer" containerID="2dbb22e6e515fbc33d7af9dac1da9b5f915f0458526f25ab5aea6a320fe3a9aa" Oct 30 22:00:05.751100 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:05.751066 4933 scope.go:122] "RemoveContainer" containerID="beb4227ea657e70d78418c5d6f65a368e77021bdabd334e4cbd021c10776335a" Oct 30 22:00:05.751552 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:05.751393 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7_odh-ai-gateway-infra(57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" Oct 30 22:00:05.752799 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:05.752775 4933 scope.go:122] "RemoveContainer" containerID="73b40df86b710e22ac0100e911018998557d9da844ad8c899907adfb0fab98c3" Oct 30 22:00:05.753061 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:05.753033 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp_odh-ai-gateway-infra(e4338b3b-be7d-4f8a-896a-5245bb9deec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" Oct 30 22:00:05.754807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:05.754786 4933 scope.go:122] "RemoveContainer" containerID="d8088c9f794b7b1c6db7246ed451d263c53763341d476d61c746bf96527df950" Oct 30 22:00:05.755053 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:05.755030 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx_odh-ai-gateway-infra(b1554472-bf3c-4941-a468-6fb08f27b445)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" Oct 30 22:00:05.756767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:05.756748 4933 scope.go:122] "RemoveContainer" containerID="602e43601d9ea0ed7834fead83d536796fdaed6f9f23684c8218c07a7667b454" Oct 30 22:00:05.756954 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:05.756936 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r_odh-ai-gateway-infra(9862cec8-1f8e-4400-924a-428883f932fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" podUID="9862cec8-1f8e-4400-924a-428883f932fc" Oct 30 22:00:14.459234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:14.459202 4933 scope.go:122] "RemoveContainer" containerID="91fb65cad619a27822e52cc952167067c2b39e8132d0975c8c00a0fbb7f325f3" Oct 30 22:00:14.832585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:14.832540 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" event={"ID":"b9a2aa35-79f7-4d31-8345-8082258e0aa0","Type":"ContainerStarted","Data":"e7467f9da933b1ec1c110927f2c0c7b535ed25086cff20c0c71a47f3d9ce3ae5"} Oct 30 22:00:15.459126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:15.459097 4933 scope.go:122] "RemoveContainer" containerID="beb4227ea657e70d78418c5d6f65a368e77021bdabd334e4cbd021c10776335a" Oct 30 22:00:15.842807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:15.842783 4933 generic.go:356] "Generic (PLEG): container finished" podID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerID="e7467f9da933b1ec1c110927f2c0c7b535ed25086cff20c0c71a47f3d9ce3ae5" exitCode=60 Oct 30 22:00:15.843190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:15.842857 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" event={"ID":"b9a2aa35-79f7-4d31-8345-8082258e0aa0","Type":"ContainerDied","Data":"e7467f9da933b1ec1c110927f2c0c7b535ed25086cff20c0c71a47f3d9ce3ae5"} Oct 30 22:00:15.843190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:15.842921 4933 scope.go:122] "RemoveContainer" containerID="91fb65cad619a27822e52cc952167067c2b39e8132d0975c8c00a0fbb7f325f3" Oct 30 22:00:15.843314 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:15.843247 4933 scope.go:122] "RemoveContainer" containerID="e7467f9da933b1ec1c110927f2c0c7b535ed25086cff20c0c71a47f3d9ce3ae5" Oct 30 22:00:15.843559 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:15.843532 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-wrsqq_odh-ai-gateway-infra(b9a2aa35-79f7-4d31-8345-8082258e0aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" Oct 30 22:00:15.845084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:15.845064 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerStarted","Data":"614a8455bb517d369988a61bfecb9d128426a9e0af8cf0d55712a3ea5d896b05"} Oct 30 22:00:15.959195 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:15.959170 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq"] Oct 30 22:00:16.459700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:16.459640 4933 scope.go:122] "RemoveContainer" containerID="d8088c9f794b7b1c6db7246ed451d263c53763341d476d61c746bf96527df950" Oct 30 22:00:16.854551 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:16.854518 4933 generic.go:356] "Generic (PLEG): container finished" podID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerID="614a8455bb517d369988a61bfecb9d128426a9e0af8cf0d55712a3ea5d896b05" exitCode=60 Oct 30 22:00:16.855008 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:16.854587 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerDied","Data":"614a8455bb517d369988a61bfecb9d128426a9e0af8cf0d55712a3ea5d896b05"} Oct 30 22:00:16.855008 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:16.854636 4933 scope.go:122] "RemoveContainer" containerID="beb4227ea657e70d78418c5d6f65a368e77021bdabd334e4cbd021c10776335a" Oct 30 22:00:16.855008 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:16.854979 4933 scope.go:122] "RemoveContainer" containerID="614a8455bb517d369988a61bfecb9d128426a9e0af8cf0d55712a3ea5d896b05" Oct 30 22:00:16.855266 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:16.855247 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7_odh-ai-gateway-infra(57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" Oct 30 22:00:16.856795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:16.856774 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerStarted","Data":"46c5981f59af326c23c28e056fd21c69c6848a6147751b10ec98d7c71cf4cb1f"} Oct 30 22:00:17.013134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.013110 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:17.103883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.103854 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a2aa35-79f7-4d31-8345-8082258e0aa0-kube-api-access-q8c2p\" (UniqueName: \"kubernetes.io/projected/b9a2aa35-79f7-4d31-8345-8082258e0aa0-kube-api-access-q8c2p\") pod \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " Oct 30 22:00:17.104051 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.103991 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a2aa35-79f7-4d31-8345-8082258e0aa0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a2aa35-79f7-4d31-8345-8082258e0aa0-maas-api-service-ca\") pod \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\" (UID: \"b9a2aa35-79f7-4d31-8345-8082258e0aa0\") " Oct 30 22:00:17.104345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.104321 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a2aa35-79f7-4d31-8345-8082258e0aa0-maas-api-service-ca" pod "b9a2aa35-79f7-4d31-8345-8082258e0aa0" (UID: "b9a2aa35-79f7-4d31-8345-8082258e0aa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:17.105621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.105578 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7"] Oct 30 22:00:17.106121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.106098 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a2aa35-79f7-4d31-8345-8082258e0aa0-kube-api-access-q8c2p" pod "b9a2aa35-79f7-4d31-8345-8082258e0aa0" (UID: "b9a2aa35-79f7-4d31-8345-8082258e0aa0"). InnerVolumeSpecName "kube-api-access-q8c2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:17.205639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.205610 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a2aa35-79f7-4d31-8345-8082258e0aa0-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:17.205639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.205640 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8c2p\" (UniqueName: \"kubernetes.io/projected/b9a2aa35-79f7-4d31-8345-8082258e0aa0-kube-api-access-q8c2p\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:17.869868 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.869836 4933 generic.go:356] "Generic (PLEG): container finished" podID="b1554472-bf3c-4941-a468-6fb08f27b445" containerID="46c5981f59af326c23c28e056fd21c69c6848a6147751b10ec98d7c71cf4cb1f" exitCode=60 Oct 30 22:00:17.870239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.869913 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerDied","Data":"46c5981f59af326c23c28e056fd21c69c6848a6147751b10ec98d7c71cf4cb1f"} Oct 30 22:00:17.870239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.869991 4933 scope.go:122] "RemoveContainer" containerID="d8088c9f794b7b1c6db7246ed451d263c53763341d476d61c746bf96527df950" Oct 30 22:00:17.870395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.870377 4933 scope.go:122] "RemoveContainer" containerID="46c5981f59af326c23c28e056fd21c69c6848a6147751b10ec98d7c71cf4cb1f" Oct 30 22:00:17.870754 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:17.870729 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx_odh-ai-gateway-infra(b1554472-bf3c-4941-a468-6fb08f27b445)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" Oct 30 22:00:17.872099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.872002 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" event={"ID":"b9a2aa35-79f7-4d31-8345-8082258e0aa0","Type":"ContainerDied","Data":"890daeb7fb56994327b3229b92e080e7f17e32b6bff8c34cd430fa22d8a5cc8c"} Oct 30 22:00:17.872099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.872014 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq" Oct 30 22:00:17.886878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:17.885869 4933 scope.go:122] "RemoveContainer" containerID="e7467f9da933b1ec1c110927f2c0c7b535ed25086cff20c0c71a47f3d9ce3ae5" Oct 30 22:00:18.032073 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.032048 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:18.098335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.098305 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq"] Oct 30 22:00:18.104108 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.104083 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-wrsqq"] Oct 30 22:00:18.115483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.115460 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-maas-api-service-ca\") pod \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " Oct 30 22:00:18.115685 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.115668 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-kube-api-access-fdn9c\" (UniqueName: \"kubernetes.io/projected/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-kube-api-access-fdn9c\") pod \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\" (UID: \"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9\") " Oct 30 22:00:18.115801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.115782 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-maas-api-service-ca" pod "57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" (UID: "57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.116057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.116041 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.117955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.117904 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-kube-api-access-fdn9c" pod "57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" (UID: "57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9"). InnerVolumeSpecName "kube-api-access-fdn9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:18.217468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.217444 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdn9c\" (UniqueName: \"kubernetes.io/projected/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9-kube-api-access-fdn9c\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.218988 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.218968 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx"] Oct 30 22:00:18.459298 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.459210 4933 scope.go:122] "RemoveContainer" containerID="73b40df86b710e22ac0100e911018998557d9da844ad8c899907adfb0fab98c3" Oct 30 22:00:18.459469 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.459335 4933 scope.go:122] "RemoveContainer" containerID="602e43601d9ea0ed7834fead83d536796fdaed6f9f23684c8218c07a7667b454" Oct 30 22:00:18.880832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.880801 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerStarted","Data":"07a8c59001669dd238434b62a66380feeea15e543fb432cbd0773aa98de1668e"} Oct 30 22:00:18.883434 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.883410 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" Oct 30 22:00:18.883434 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.883418 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7" event={"ID":"57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9","Type":"ContainerDied","Data":"51b454a6897b942c9bc8d4210d1195674f1e248dd329ec378c3f5b54d09c978f"} Oct 30 22:00:18.883640 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.883471 4933 scope.go:122] "RemoveContainer" containerID="614a8455bb517d369988a61bfecb9d128426a9e0af8cf0d55712a3ea5d896b05" Oct 30 22:00:18.885546 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:18.885522 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerStarted","Data":"73b192952c4ddcadbdb02674324078d6921f1258e4a490ece0df0f4e2229f47b"} Oct 30 22:00:19.057809 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.057781 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:19.128310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.128283 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1554472-bf3c-4941-a468-6fb08f27b445-kube-api-access-mchfm\" (UniqueName: \"kubernetes.io/projected/b1554472-bf3c-4941-a468-6fb08f27b445-kube-api-access-mchfm\") pod \"b1554472-bf3c-4941-a468-6fb08f27b445\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " Oct 30 22:00:19.128459 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.128391 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1554472-bf3c-4941-a468-6fb08f27b445-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1554472-bf3c-4941-a468-6fb08f27b445-maas-api-service-ca\") pod \"b1554472-bf3c-4941-a468-6fb08f27b445\" (UID: \"b1554472-bf3c-4941-a468-6fb08f27b445\") " Oct 30 22:00:19.128774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.128753 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1554472-bf3c-4941-a468-6fb08f27b445-maas-api-service-ca" pod "b1554472-bf3c-4941-a468-6fb08f27b445" (UID: "b1554472-bf3c-4941-a468-6fb08f27b445"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:19.133638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.133602 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1554472-bf3c-4941-a468-6fb08f27b445-kube-api-access-mchfm" pod "b1554472-bf3c-4941-a468-6fb08f27b445" (UID: "b1554472-bf3c-4941-a468-6fb08f27b445"). InnerVolumeSpecName "kube-api-access-mchfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:19.229456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.229428 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mchfm\" (UniqueName: \"kubernetes.io/projected/b1554472-bf3c-4941-a468-6fb08f27b445-kube-api-access-mchfm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.229456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.229449 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1554472-bf3c-4941-a468-6fb08f27b445-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:19.348790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.348465 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7"] Oct 30 22:00:19.354872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.354846 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b0c55f-29889960-zs2s7"] Oct 30 22:00:19.470509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.469548 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" path="/var/lib/kubelet/pods/57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9/volumes" Oct 30 22:00:19.470509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.470111 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" path="/var/lib/kubelet/pods/b9a2aa35-79f7-4d31-8345-8082258e0aa0/volumes" Oct 30 22:00:19.899785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.899751 4933 generic.go:356] "Generic (PLEG): container finished" podID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerID="73b192952c4ddcadbdb02674324078d6921f1258e4a490ece0df0f4e2229f47b" exitCode=60 Oct 30 22:00:19.900274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.899829 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerDied","Data":"73b192952c4ddcadbdb02674324078d6921f1258e4a490ece0df0f4e2229f47b"} Oct 30 22:00:19.900274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.899951 4933 scope.go:122] "RemoveContainer" containerID="73b40df86b710e22ac0100e911018998557d9da844ad8c899907adfb0fab98c3" Oct 30 22:00:19.900410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.900328 4933 scope.go:122] "RemoveContainer" containerID="73b192952c4ddcadbdb02674324078d6921f1258e4a490ece0df0f4e2229f47b" Oct 30 22:00:19.900704 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:19.900680 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp_odh-ai-gateway-infra(e4338b3b-be7d-4f8a-896a-5245bb9deec4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" Oct 30 22:00:19.901971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.901953 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" Oct 30 22:00:19.902129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.902093 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx" event={"ID":"b1554472-bf3c-4941-a468-6fb08f27b445","Type":"ContainerDied","Data":"20c8553865d8c886348fc863cebb9092325968c3115e27cd376d3ac5adfb3f7b"} Oct 30 22:00:19.904811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.904790 4933 generic.go:356] "Generic (PLEG): container finished" podID="9862cec8-1f8e-4400-924a-428883f932fc" containerID="07a8c59001669dd238434b62a66380feeea15e543fb432cbd0773aa98de1668e" exitCode=60 Oct 30 22:00:19.904811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.904815 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerDied","Data":"07a8c59001669dd238434b62a66380feeea15e543fb432cbd0773aa98de1668e"} Oct 30 22:00:19.905135 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.905122 4933 scope.go:122] "RemoveContainer" containerID="07a8c59001669dd238434b62a66380feeea15e543fb432cbd0773aa98de1668e" Oct 30 22:00:19.905329 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:19.905312 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r_odh-ai-gateway-infra(9862cec8-1f8e-4400-924a-428883f932fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" podUID="9862cec8-1f8e-4400-924a-428883f932fc" Oct 30 22:00:19.915468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.915448 4933 scope.go:122] "RemoveContainer" containerID="46c5981f59af326c23c28e056fd21c69c6848a6147751b10ec98d7c71cf4cb1f" Oct 30 22:00:19.926849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:19.926833 4933 scope.go:122] "RemoveContainer" containerID="602e43601d9ea0ed7834fead83d536796fdaed6f9f23684c8218c07a7667b454" Oct 30 22:00:20.010393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:20.010368 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r"] Oct 30 22:00:20.125918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:20.125889 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp"] Oct 30 22:00:20.134133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:20.134106 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx"] Oct 30 22:00:20.137986 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:20.137962 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-29cbad-29889960-wbltx"] Oct 30 22:00:21.133338 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.133316 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:21.135455 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.135436 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:21.254358 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.254286 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4338b3b-be7d-4f8a-896a-5245bb9deec4-kube-api-access-75dzl\" (UniqueName: \"kubernetes.io/projected/e4338b3b-be7d-4f8a-896a-5245bb9deec4-kube-api-access-75dzl\") pod \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " Oct 30 22:00:21.254520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.254420 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9862cec8-1f8e-4400-924a-428883f932fc-kube-api-access-fdgtd\" (UniqueName: \"kubernetes.io/projected/9862cec8-1f8e-4400-924a-428883f932fc-kube-api-access-fdgtd\") pod \"9862cec8-1f8e-4400-924a-428883f932fc\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " Oct 30 22:00:21.254520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.254473 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9862cec8-1f8e-4400-924a-428883f932fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9862cec8-1f8e-4400-924a-428883f932fc-maas-api-service-ca\") pod \"9862cec8-1f8e-4400-924a-428883f932fc\" (UID: \"9862cec8-1f8e-4400-924a-428883f932fc\") " Oct 30 22:00:21.254642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.254615 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4338b3b-be7d-4f8a-896a-5245bb9deec4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4338b3b-be7d-4f8a-896a-5245bb9deec4-maas-api-service-ca\") pod \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\" (UID: \"e4338b3b-be7d-4f8a-896a-5245bb9deec4\") " Oct 30 22:00:21.255061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.254870 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9862cec8-1f8e-4400-924a-428883f932fc-maas-api-service-ca" pod "9862cec8-1f8e-4400-924a-428883f932fc" (UID: "9862cec8-1f8e-4400-924a-428883f932fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.255061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.255031 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4338b3b-be7d-4f8a-896a-5245bb9deec4-maas-api-service-ca" pod "e4338b3b-be7d-4f8a-896a-5245bb9deec4" (UID: "e4338b3b-be7d-4f8a-896a-5245bb9deec4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:21.255061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.255039 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9862cec8-1f8e-4400-924a-428883f932fc-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.257221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.257193 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4338b3b-be7d-4f8a-896a-5245bb9deec4-kube-api-access-75dzl" pod "e4338b3b-be7d-4f8a-896a-5245bb9deec4" (UID: "e4338b3b-be7d-4f8a-896a-5245bb9deec4"). InnerVolumeSpecName "kube-api-access-75dzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.257332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.257285 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9862cec8-1f8e-4400-924a-428883f932fc-kube-api-access-fdgtd" pod "9862cec8-1f8e-4400-924a-428883f932fc" (UID: "9862cec8-1f8e-4400-924a-428883f932fc"). InnerVolumeSpecName "kube-api-access-fdgtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.355848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.355819 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4338b3b-be7d-4f8a-896a-5245bb9deec4-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.355848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.355851 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75dzl\" (UniqueName: \"kubernetes.io/projected/e4338b3b-be7d-4f8a-896a-5245bb9deec4-kube-api-access-75dzl\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.356023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.355867 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdgtd\" (UniqueName: \"kubernetes.io/projected/9862cec8-1f8e-4400-924a-428883f932fc-kube-api-access-fdgtd\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.469672 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.469637 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" path="/var/lib/kubelet/pods/b1554472-bf3c-4941-a468-6fb08f27b445/volumes" Oct 30 22:00:21.615060 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.615023 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:00:21.927161 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.927072 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" Oct 30 22:00:21.927161 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.927116 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r" event={"ID":"9862cec8-1f8e-4400-924a-428883f932fc","Type":"ContainerDied","Data":"60b47e7bc5db048d4c35a9d04280029b69569baa4ab5a445e63d6940e52656c7"} Oct 30 22:00:21.927389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.927187 4933 scope.go:122] "RemoveContainer" containerID="07a8c59001669dd238434b62a66380feeea15e543fb432cbd0773aa98de1668e" Oct 30 22:00:21.929275 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.929240 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" event={"ID":"e4338b3b-be7d-4f8a-896a-5245bb9deec4","Type":"ContainerDied","Data":"54c6395d26b83d3f31985915341864529d74472b1b2a260d355c7a6c469807bb"} Oct 30 22:00:21.929275 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.929270 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp" Oct 30 22:00:21.938453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:21.938434 4933 scope.go:122] "RemoveContainer" containerID="73b192952c4ddcadbdb02674324078d6921f1258e4a490ece0df0f4e2229f47b" Oct 30 22:00:22.095929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.095893 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r"] Oct 30 22:00:22.100344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.100316 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889960-wqw4r"] Oct 30 22:00:22.170641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.170613 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk"] Oct 30 22:00:22.172205 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.172164 4933 cpu_manager.go:414] "containerMap: removing container" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.172419 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.172390 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.172559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.172545 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.172692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.172677 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.172826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.172814 4933 cpu_manager.go:414] "containerMap: removing container" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.172931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.172920 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.173054 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173041 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.173265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173252 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.173367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173357 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.173461 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173450 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.173603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173590 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.173715 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173703 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.173828 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173816 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.173930 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.173918 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.174039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174028 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.174137 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174124 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.174240 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174229 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.174330 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174319 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.174431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174421 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.174534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174523 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.174667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174655 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.174768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.174757 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175044 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175058 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175066 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175074 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175086 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175095 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175105 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175113 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175126 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.177867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.175136 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.219627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.215826 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.233643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.233611 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk"] Oct 30 22:00:22.236545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.236522 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp"] Oct 30 22:00:22.248058 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.247996 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-dde1e8-29889960-9vhvp"] Oct 30 22:00:22.270666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.270641 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"distinct-w1-d516c4-kserve-self-signed-certs\"" Oct 30 22:00:22.277151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.277131 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-dshm\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.277260 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.277164 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tmp-dir\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.277323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.277271 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tls-certs\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.277323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.277306 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwjw\" (UniqueName: \"kubernetes.io/projected/f35b6179-b260-4be8-a5f2-0f94da37e2c9-kube-api-access-jxwjw\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.277559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.277526 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-model-cache\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.277678 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.277607 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-home\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379205 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tls-certs\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379257 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwjw\" (UniqueName: \"kubernetes.io/projected/f35b6179-b260-4be8-a5f2-0f94da37e2c9-kube-api-access-jxwjw\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379340 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-model-cache\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379386 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-home\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379483 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-dshm\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379554 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tmp-dir\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379808 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379786 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-home\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.379978 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379960 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tmp-dir\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.380038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.379994 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-model-cache\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.381944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.381917 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-dshm\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.382414 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.382398 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tls-certs\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.437296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.437225 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwjw\" (UniqueName: \"kubernetes.io/projected/f35b6179-b260-4be8-a5f2-0f94da37e2c9-kube-api-access-jxwjw\") pod \"distinct-w1-d516c4-kserve-9b8955b4c-xx2mk\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.521827 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.521791 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22"] Oct 30 22:00:22.522627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522603 4933 cpu_manager.go:414] "containerMap: removing container" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.522627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522625 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.522826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522648 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.522826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522658 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.522826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522675 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.522826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522683 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.522826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522695 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.522826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522710 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.523118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522887 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a2aa35-79f7-4d31-8345-8082258e0aa0" containerName="cleanup" Oct 30 22:00:22.523118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522902 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1554472-bf3c-4941-a468-6fb08f27b445" containerName="cleanup" Oct 30 22:00:22.523118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522912 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9862cec8-1f8e-4400-924a-428883f932fc" containerName="cleanup" Oct 30 22:00:22.523118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522924 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fe3ec0-da54-4ecb-bdb2-6c04a339e6d9" containerName="cleanup" Oct 30 22:00:22.523118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.522935 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" containerName="cleanup" Oct 30 22:00:22.529698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.529679 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.550014 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.549989 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:22.584339 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.584310 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22"] Oct 30 22:00:22.589410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.589382 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-0c99b1-openshift-default-dockercfg-r8hwd\"" Oct 30 22:00:22.688343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688018 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688102 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688132 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688209 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688270 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688429 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688483 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjcz\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-kube-api-access-7fjcz\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688550 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d841f661-326b-4262-9765-e406ee031b58-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.688736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.688579 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d841f661-326b-4262-9765-e406ee031b58-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790065 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjcz\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-kube-api-access-7fjcz\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790108 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d841f661-326b-4262-9765-e406ee031b58-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790126 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d841f661-326b-4262-9765-e406ee031b58-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790211 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790258 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790280 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790344 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790402 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790439 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790694 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.790834 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790810 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.791000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.790974 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d841f661-326b-4262-9765-e406ee031b58-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.791152 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.791104 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.791344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.791323 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.792802 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.792784 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.792860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.792845 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d841f661-326b-4262-9765-e406ee031b58-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.852762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.852731 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.853078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.853052 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjcz\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-kube-api-access-7fjcz\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:22.978997 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:00:22.978970 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf35b6179_b260_4be8_a5f2_0f94da37e2c9.slice/crio-2a948a258d204ed3d837bdb365446cb49493ed0f0341ec92615fb15040cb0cb1 WatchSource:0}: Error finding container 2a948a258d204ed3d837bdb365446cb49493ed0f0341ec92615fb15040cb0cb1: Status 404 returned error can't find the container with id 2a948a258d204ed3d837bdb365446cb49493ed0f0341ec92615fb15040cb0cb1 Oct 30 22:00:22.979630 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:22.979605 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk"] Oct 30 22:00:23.152051 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.152022 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:23.466361 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.466333 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9862cec8-1f8e-4400-924a-428883f932fc" path="/var/lib/kubelet/pods/9862cec8-1f8e-4400-924a-428883f932fc/volumes" Oct 30 22:00:23.466776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.466752 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4338b3b-be7d-4f8a-896a-5245bb9deec4" path="/var/lib/kubelet/pods/e4338b3b-be7d-4f8a-896a-5245bb9deec4/volumes" Oct 30 22:00:23.579053 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:00:23.579022 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd841f661_326b_4262_9765_e406ee031b58.slice/crio-6661a20fff01bc0ecf44d92dfe6fcfcabde393eb5dde93821575529a52d19438 WatchSource:0}: Error finding container 6661a20fff01bc0ecf44d92dfe6fcfcabde393eb5dde93821575529a52d19438: Status 404 returned error can't find the container with id 6661a20fff01bc0ecf44d92dfe6fcfcabde393eb5dde93821575529a52d19438 Oct 30 22:00:23.579401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.579384 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22"] Oct 30 22:00:23.581497 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.581456 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:23.581575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.581552 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:23.581613 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.581586 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:23.950832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.950792 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" event={"ID":"d841f661-326b-4262-9765-e406ee031b58","Type":"ContainerStarted","Data":"1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef"} Oct 30 22:00:23.951023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.950854 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" event={"ID":"d841f661-326b-4262-9765-e406ee031b58","Type":"ContainerStarted","Data":"6661a20fff01bc0ecf44d92dfe6fcfcabde393eb5dde93821575529a52d19438"} Oct 30 22:00:23.952720 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.952696 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" event={"ID":"f35b6179-b260-4be8-a5f2-0f94da37e2c9","Type":"ContainerStarted","Data":"9277b773c79160900ff31206cdc2288f2f27102e64bf7331125082bd2ea46990"} Oct 30 22:00:23.952822 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:23.952728 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" event={"ID":"f35b6179-b260-4be8-a5f2-0f94da37e2c9","Type":"ContainerStarted","Data":"2a948a258d204ed3d837bdb365446cb49493ed0f0341ec92615fb15040cb0cb1"} Oct 30 22:00:24.071173 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:24.071111 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podStartSLOduration=2.07109804 podStartE2EDuration="2.07109804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:24.069146318 +0000 UTC m=+1072.750296212" watchObservedRunningTime="2026-10-30 22:00:24.07109804 +0000 UTC m=+1072.752247932" Oct 30 22:00:24.154965 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:24.154870 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:24.159942 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:24.159451 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:00:24.159942 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:24.159534 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:00:24.191255 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:24.191205 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podStartSLOduration=2.191189829 podStartE2EDuration="2.191189829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:24.187823038 +0000 UTC m=+1072.868972957" watchObservedRunningTime="2026-10-30 22:00:24.191189829 +0000 UTC m=+1072.872339721" Oct 30 22:00:24.555597 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:24.555564 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:25.155308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:25.154868 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:00:25.155308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:25.154927 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:00:26.165900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:26.165863 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:00:26.166483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:26.166449 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:00:27.152977 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:27.152940 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:00:27.153300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:27.153272 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:00:28.158515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:28.157871 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:00:28.158515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:28.157969 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:00:28.573570 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:28.573529 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:00:29.159904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:29.159870 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 22:00:29.160541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:29.160469 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 22:00:30.277289 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:30.277260 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:30.278031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:30.278005 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:30.278945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:30.278924 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:00:32.551183 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:32.551143 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:32.563700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:32.563679 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:33.037166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:33.037144 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:00:54.796366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.796328 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6"] Oct 30 22:00:54.800234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.800214 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.853575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.853548 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-dde1e8-openshift-default-dockercfg-5l6vx\"" Oct 30 22:00:54.886739 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.886713 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.886839 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.886747 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.886891 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.886869 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.886932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.886902 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.886986 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.886972 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm9h\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-kube-api-access-fnm9h\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.887068 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.887054 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132ebc6-e825-4a81-9048-0119ce05028c-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.887103 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.887087 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.887138 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.887113 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.887192 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.887179 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132ebc6-e825-4a81-9048-0119ce05028c-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.911587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.911559 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6"] Oct 30 22:00:54.988184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988149 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132ebc6-e825-4a81-9048-0119ce05028c-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988198 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988383 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988222 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988383 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988239 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132ebc6-e825-4a81-9048-0119ce05028c-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988383 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988283 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988383 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988302 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988398 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988425 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988483 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm9h\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-kube-api-access-fnm9h\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988636 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988728 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988779 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.988913 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.988889 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.989194 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.989170 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132ebc6-e825-4a81-9048-0119ce05028c-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.990661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.990643 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:54.991049 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:54.991034 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132ebc6-e825-4a81-9048-0119ce05028c-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:55.046730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:55.046670 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:55.046824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:55.046794 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm9h\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-kube-api-access-fnm9h\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:55.115533 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:55.115485 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:55.544597 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:00:55.544559 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1132ebc6_e825_4a81_9048_0119ce05028c.slice/crio-23aed8148af31d66ba02743a5be183235b8602dab7e2f3184c710761e86e21bc WatchSource:0}: Error finding container 23aed8148af31d66ba02743a5be183235b8602dab7e2f3184c710761e86e21bc: Status 404 returned error can't find the container with id 23aed8148af31d66ba02743a5be183235b8602dab7e2f3184c710761e86e21bc Oct 30 22:00:55.545458 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:55.545431 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6"] Oct 30 22:00:55.547181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:55.547147 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:55.547271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:55.547211 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:55.547271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:55.547257 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:00:56.219877 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:56.219837 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" event={"ID":"1132ebc6-e825-4a81-9048-0119ce05028c","Type":"ContainerStarted","Data":"cd12e4d0d6ab5d1c37a27109ed9cedc7083e3ad4850be3aa3b9a273854607acc"} Oct 30 22:00:56.220388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:56.219897 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" event={"ID":"1132ebc6-e825-4a81-9048-0119ce05028c","Type":"ContainerStarted","Data":"23aed8148af31d66ba02743a5be183235b8602dab7e2f3184c710761e86e21bc"} Oct 30 22:00:56.347214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:56.347116 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podStartSLOduration=2.3470952880000002 podStartE2EDuration="2.347095288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:56.341683686 +0000 UTC m=+1105.022833578" watchObservedRunningTime="2026-10-30 22:00:56.347095288 +0000 UTC m=+1105.028245196" Oct 30 22:00:57.116581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:57.116546 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:00:57.117913 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:57.117885 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:00:57.118047 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:57.117929 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:00:58.116317 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:58.116284 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:00:58.116786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:58.116339 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:00:58.570223 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:00:58.570186 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:00:59.116504 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:59.116460 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:00:59.116908 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:00:59.116533 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:01:00.116539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:00.116503 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:01:00.116948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:00.116556 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:01:01.116528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:01.116468 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:01:01.116722 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:01.116551 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:01:02.115802 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:02.115765 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:01:02.116007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:02.115828 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 22:01:03.120984 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:03.120953 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:01:03.121474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:03.121375 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:01:03.122075 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:03.122055 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:01:09.272324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.272287 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th"] Oct 30 22:01:09.289824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.289794 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.334753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.334713 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th"] Oct 30 22:01:09.398152 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398114 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398337 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398164 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadd75e2-49fd-4adb-9128-d185937f4bc1-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398337 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398194 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398337 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398218 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5mk\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-kube-api-access-pn5mk\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398337 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398322 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398449 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398474 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398521 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.398671 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.398645 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500382 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500598 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500469 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500598 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500533 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500598 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500556 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadd75e2-49fd-4adb-9128-d185937f4bc1-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500598 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500575 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500598 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500593 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5mk\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-kube-api-access-pn5mk\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500634 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500698 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.500872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500743 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.501016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500882 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.501016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500907 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.501016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.500997 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.501214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.501193 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.501384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.501363 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadd75e2-49fd-4adb-9128-d185937f4bc1-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.503204 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.503180 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.503415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.503397 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.562720 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.562689 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.562887 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.562863 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5mk\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-kube-api-access-pn5mk\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:09.614520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:09.614480 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:10.054425 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:01:10.054392 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadd75e2_49fd_4adb_9128_d185937f4bc1.slice/crio-70b5bfe5c2fbde9403769c3bcaae420e0d1215efc0038f215972af0567eca5e9 WatchSource:0}: Error finding container 70b5bfe5c2fbde9403769c3bcaae420e0d1215efc0038f215972af0567eca5e9: Status 404 returned error can't find the container with id 70b5bfe5c2fbde9403769c3bcaae420e0d1215efc0038f215972af0567eca5e9 Oct 30 22:01:10.055610 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.055584 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th"] Oct 30 22:01:10.057258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.057219 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:10.057338 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.057303 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:10.057377 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.057345 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:10.366712 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.366626 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" event={"ID":"cadd75e2-49fd-4adb-9128-d185937f4bc1","Type":"ContainerStarted","Data":"0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00"} Oct 30 22:01:10.367277 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.367244 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" event={"ID":"cadd75e2-49fd-4adb-9128-d185937f4bc1","Type":"ContainerStarted","Data":"70b5bfe5c2fbde9403769c3bcaae420e0d1215efc0038f215972af0567eca5e9"} Oct 30 22:01:10.548237 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.548133 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podStartSLOduration=1.5481136420000001 podStartE2EDuration="1.548113642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:10.49127701 +0000 UTC m=+1119.172426915" watchObservedRunningTime="2026-10-30 22:01:10.548113642 +0000 UTC m=+1119.229263536" Oct 30 22:01:10.614676 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.614644 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:10.663615 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:10.663521 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:01:11.616843 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:11.616810 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:01:11.617241 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:11.616871 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 22:01:12.615219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:12.615178 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:01:12.615395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:12.615254 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 22:01:13.468526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.468478 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx"] Oct 30 22:01:13.472696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.472676 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.529217 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.529187 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx"] Oct 30 22:01:13.563951 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.563921 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6xj\" (UniqueName: \"kubernetes.io/projected/91d61953-e352-44cd-9276-8285a2e665dc-kube-api-access-pm6xj\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.564094 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.563960 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d61953-e352-44cd-9276-8285a2e665dc-tls-certs\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.564094 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.564087 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-model-cache\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.564197 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.564106 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-home\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.564197 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.564125 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-tmp-dir\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.564197 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.564155 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-dshm\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.572723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.572702 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9f8aea\"/\"distinct-2-w5-9f8aea-kserve-self-signed-certs\"" Oct 30 22:01:13.615049 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.615016 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:01:13.615219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.615070 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 22:01:13.666369 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.666325 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-model-cache\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.667218 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.667183 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-home\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.667787 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.667769 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-tmp-dir\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.667905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.667830 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-dshm\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.667976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.667904 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6xj\" (UniqueName: \"kubernetes.io/projected/91d61953-e352-44cd-9276-8285a2e665dc-kube-api-access-pm6xj\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.667976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.667943 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d61953-e352-44cd-9276-8285a2e665dc-tls-certs\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.667976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.667712 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-home\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.668147 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.667023 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-model-cache\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.668996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.668974 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-tmp-dir\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.680153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.680118 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d61953-e352-44cd-9276-8285a2e665dc-tls-certs\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.688086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.688057 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-dshm\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.768480 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.768398 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6xj\" (UniqueName: \"kubernetes.io/projected/91d61953-e352-44cd-9276-8285a2e665dc-kube-api-access-pm6xj\") pod \"distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:13.785581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:13.785544 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:14.303535 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:01:14.303465 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91d61953_e352_44cd_9276_8285a2e665dc.slice/crio-b7b2916fff8dc6057453173e350a1a0da15289c33ce62ae477af83f7efece7e2 WatchSource:0}: Error finding container b7b2916fff8dc6057453173e350a1a0da15289c33ce62ae477af83f7efece7e2: Status 404 returned error can't find the container with id b7b2916fff8dc6057453173e350a1a0da15289c33ce62ae477af83f7efece7e2 Oct 30 22:01:14.304594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:14.304563 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx"] Oct 30 22:01:14.400122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:14.400095 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" event={"ID":"91d61953-e352-44cd-9276-8285a2e665dc","Type":"ContainerStarted","Data":"b7b2916fff8dc6057453173e350a1a0da15289c33ce62ae477af83f7efece7e2"} Oct 30 22:01:14.615507 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:14.615409 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:01:14.615507 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:14.615466 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 22:01:15.409382 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:15.409344 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" event={"ID":"91d61953-e352-44cd-9276-8285a2e665dc","Type":"ContainerStarted","Data":"7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50"} Oct 30 22:01:15.528927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:15.528878 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podStartSLOduration=2.528864381 podStartE2EDuration="2.528864381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:15.525159298 +0000 UTC m=+1124.206309192" watchObservedRunningTime="2026-10-30 22:01:15.528864381 +0000 UTC m=+1124.210014274" Oct 30 22:01:15.615765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:15.615731 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:01:15.616172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:15.615782 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 22:01:15.786300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:15.786266 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:16.614891 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:16.614859 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 22:01:16.615114 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:16.614919 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 22:01:17.619663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:17.619633 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:17.620168 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:17.620098 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:17.620818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:17.620797 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:01:23.786953 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:23.786915 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:23.799296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:23.799272 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:24.489986 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:24.489956 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:01:28.547927 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:01:28.547889 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:01:39.818876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.818848 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q"] Oct 30 22:01:39.824202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.824184 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.877475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.877449 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0f42cb-openshift-default-dockercfg-7vjjp\"" Oct 30 22:01:39.926309 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.926275 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.926475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.926378 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.926475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.926408 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.927744 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.927713 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-kube-api-access-tfjxv\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.927882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.927760 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.927882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.927850 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.928023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.927891 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.928023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.927945 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.928136 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.928050 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:39.934885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:39.934862 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q"] Oct 30 22:01:40.028863 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.028833 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.028888 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.028914 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.028945 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-kube-api-access-tfjxv\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.028971 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029001 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029035 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029093 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029215 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029243 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029292 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029383 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029700 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.029928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.029905 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.031615 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.031588 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.031977 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.031952 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.087642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.087573 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.087804 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.087780 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-kube-api-access-tfjxv\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.138148 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.138110 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:40.579941 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.579900 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q"] Oct 30 22:01:40.580114 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:01:40.580087 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ddd21c_c122_4046_9804_c3bcd2b34bc3.slice/crio-8e93f49494c41224544fb6d4529aed2e47b7f1558eca88f49cb404b790892f8b WatchSource:0}: Error finding container 8e93f49494c41224544fb6d4529aed2e47b7f1558eca88f49cb404b790892f8b: Status 404 returned error can't find the container with id 8e93f49494c41224544fb6d4529aed2e47b7f1558eca88f49cb404b790892f8b Oct 30 22:01:40.582607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.582574 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:40.582696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.582638 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:40.582696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.582668 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:01:40.613960 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:40.613939 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" event={"ID":"d3ddd21c-c122-4046-9804-c3bcd2b34bc3","Type":"ContainerStarted","Data":"8e93f49494c41224544fb6d4529aed2e47b7f1558eca88f49cb404b790892f8b"} Oct 30 22:01:41.622795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:41.622758 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" event={"ID":"d3ddd21c-c122-4046-9804-c3bcd2b34bc3","Type":"ContainerStarted","Data":"237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b"} Oct 30 22:01:41.793875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:41.793819 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podStartSLOduration=2.79380485 podStartE2EDuration="2.79380485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:41.791302536 +0000 UTC m=+1150.472452429" watchObservedRunningTime="2026-10-30 22:01:41.79380485 +0000 UTC m=+1150.474954743" Oct 30 22:01:42.142039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:42.142009 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:42.143189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:42.143164 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:01:42.143395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:42.143368 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:01:43.140837 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:43.140798 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:01:43.141424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:43.141390 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:01:44.139825 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:44.139791 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:01:44.140118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:44.140091 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:01:45.139835 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:45.139795 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:01:45.140436 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:45.140398 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:01:46.139521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:46.138860 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:01:46.139521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:46.138930 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:01:47.139503 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:47.139448 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:01:47.139987 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:47.139561 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 22:01:48.142368 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:48.142337 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:48.142871 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:48.142841 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:48.143663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:48.143643 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:01:49.559944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:01:49.559915 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:01:58.544596 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:01:58.544545 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:02:01.895635 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:01.895604 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb"] Oct 30 22:02:01.899844 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:01.899818 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:01.957391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:01.957366 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb"] Oct 30 22:02:02.000406 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.000377 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvmzh\" (UniqueName: \"kubernetes.io/projected/bc7decc5-01c0-4583-b540-b5ab1461d0d1-kube-api-access-nvmzh\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.000565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.000421 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.000655 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.000635 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.001629 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.001613 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:02.001733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.001642 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:02.001733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.001655 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:02.003137 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.003122 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:02:02.101477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.101447 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.101642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.101598 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvmzh\" (UniqueName: \"kubernetes.io/projected/bc7decc5-01c0-4583-b540-b5ab1461d0d1-kube-api-access-nvmzh\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.101705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.101646 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.102093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.102072 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.104332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.104304 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.159714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.159650 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvmzh\" (UniqueName: \"kubernetes.io/projected/bc7decc5-01c0-4583-b540-b5ab1461d0d1-kube-api-access-nvmzh\") pod \"maas-api-revoke-sub-f4ca3806-vvtdb\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.214619 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.214597 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:02.594420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.593664 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb"] Oct 30 22:02:02.594420 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:02:02.593713 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7decc5_01c0_4583_b540_b5ab1461d0d1.slice/crio-b8f0af3cdafe6271e16916dfb9d030b8f1b5a81e4b43e2ee4e95f87812d1e822 WatchSource:0}: Error finding container b8f0af3cdafe6271e16916dfb9d030b8f1b5a81e4b43e2ee4e95f87812d1e822: Status 404 returned error can't find the container with id b8f0af3cdafe6271e16916dfb9d030b8f1b5a81e4b43e2ee4e95f87812d1e822 Oct 30 22:02:02.828261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:02.828234 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" event={"ID":"bc7decc5-01c0-4583-b540-b5ab1461d0d1","Type":"ContainerStarted","Data":"b8f0af3cdafe6271e16916dfb9d030b8f1b5a81e4b43e2ee4e95f87812d1e822"} Oct 30 22:02:03.837670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:03.837560 4933 generic.go:356] "Generic (PLEG): container finished" podID="bc7decc5-01c0-4583-b540-b5ab1461d0d1" containerID="d68505ee3bd51d406dbef601fffe7fb9c14332c38b671249695d359e389e6923" exitCode=0 Oct 30 22:02:03.838331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:03.838266 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" event={"ID":"bc7decc5-01c0-4583-b540-b5ab1461d0d1","Type":"ContainerDied","Data":"d68505ee3bd51d406dbef601fffe7fb9c14332c38b671249695d359e389e6923"} Oct 30 22:02:05.007504 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.007469 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:05.137792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.137709 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-xks-ca\") pod \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " Oct 30 22:02:05.138139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.138106 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7decc5-01c0-4583-b540-b5ab1461d0d1-kube-api-access-nvmzh\" (UniqueName: \"kubernetes.io/projected/bc7decc5-01c0-4583-b540-b5ab1461d0d1-kube-api-access-nvmzh\") pod \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " Oct 30 22:02:05.138279 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.138269 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-openshift-ca\") pod \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\" (UID: \"bc7decc5-01c0-4583-b540-b5ab1461d0d1\") " Oct 30 22:02:05.139717 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.139686 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-openshift-ca" pod "bc7decc5-01c0-4583-b540-b5ab1461d0d1" (UID: "bc7decc5-01c0-4583-b540-b5ab1461d0d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:05.141686 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.141647 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-xks-ca" pod "bc7decc5-01c0-4583-b540-b5ab1461d0d1" (UID: "bc7decc5-01c0-4583-b540-b5ab1461d0d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:05.142184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.142159 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7decc5-01c0-4583-b540-b5ab1461d0d1-kube-api-access-nvmzh" pod "bc7decc5-01c0-4583-b540-b5ab1461d0d1" (UID: "bc7decc5-01c0-4583-b540-b5ab1461d0d1"). InnerVolumeSpecName "kube-api-access-nvmzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:05.240468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.240426 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:05.240468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.240467 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvmzh\" (UniqueName: \"kubernetes.io/projected/bc7decc5-01c0-4583-b540-b5ab1461d0d1-kube-api-access-nvmzh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:05.240733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.240504 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7decc5-01c0-4583-b540-b5ab1461d0d1-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:05.855282 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.855252 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" Oct 30 22:02:05.855454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.855290 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb" event={"ID":"bc7decc5-01c0-4583-b540-b5ab1461d0d1","Type":"ContainerDied","Data":"b8f0af3cdafe6271e16916dfb9d030b8f1b5a81e4b43e2ee4e95f87812d1e822"} Oct 30 22:02:05.855454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:05.855337 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f0af3cdafe6271e16916dfb9d030b8f1b5a81e4b43e2ee4e95f87812d1e822" Oct 30 22:02:19.062391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:19.062358 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq"] Oct 30 22:02:19.112668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:19.112642 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37edb3a7-65wvq"] Oct 30 22:02:19.466454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:19.466373 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d11304-65d3-488c-a40a-05190692e976" path="/var/lib/kubelet/pods/10d11304-65d3-488c-a40a-05190692e976/volumes" Oct 30 22:02:23.854013 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.853982 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x"] Oct 30 22:02:23.854559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.854543 4933 cpu_manager.go:414] "containerMap: removing container" podUID="bc7decc5-01c0-4583-b540-b5ab1461d0d1" containerName="revoke-keys" Oct 30 22:02:23.854559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.854558 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7decc5-01c0-4583-b540-b5ab1461d0d1" containerName="revoke-keys" Oct 30 22:02:23.854666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.854645 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7decc5-01c0-4583-b540-b5ab1461d0d1" containerName="revoke-keys" Oct 30 22:02:23.858346 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.858328 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.904476 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.904452 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb"] Oct 30 22:02:23.908374 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.908341 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:23.953037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953005 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953082 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953229 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953183 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953289 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953261 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953406 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953441 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953464 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgnc5\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-kube-api-access-kgnc5\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953618 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953585 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.953698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.953666 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:23.961352 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:23.961325 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x"] Oct 30 22:02:24.019921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.019890 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb"] Oct 30 22:02:24.055102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055060 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055108 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055133 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgnc5\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-kube-api-access-kgnc5\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055192 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh59r\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-kube-api-access-fh59r\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.055269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055222 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.055269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055260 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055313 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055345 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.055593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055373 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.055593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055409 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26975667-9851-4f05-a4ff-a9ea7e08c769-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.055593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055435 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.055593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055462 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.055593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055517 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055603 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055656 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055834 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.055950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055926 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.056210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.055961 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.056210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.056006 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.056210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.056065 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.056210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.056202 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.056651 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.056630 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.057127 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.057106 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.058843 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.058801 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.060720 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.060695 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.113278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.113218 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgnc5\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-kube-api-access-kgnc5\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.113467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.113448 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.157188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157151 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157185 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157209 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26975667-9851-4f05-a4ff-a9ea7e08c769-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157233 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157262 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157338 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157381 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157660 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157574 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh59r\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-kube-api-access-fh59r\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157660 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157607 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.157660 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157609 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.158156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157824 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-data\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.158156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157859 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-credential-socket\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.158156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157948 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26975667-9851-4f05-a4ff-a9ea7e08c769-istiod-ca-cert\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.158156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.157986 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-certs\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.160099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.160078 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-podinfo\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.160287 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.160263 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-envoy\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.172329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.172307 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:24.215289 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.215251 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-token\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.215444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.215399 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh59r\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-kube-api-access-fh59r\") pod \"e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.222229 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.222204 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:24.604572 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:02:24.604535 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcaa605_fa45_4f88_ac6f_12a4d9b4363b.slice/crio-0f09906fcdd1ef8d2b620d50c6f6ae105e8f3ff9a4ba34be9dcb81b4f5bb0bc7 WatchSource:0}: Error finding container 0f09906fcdd1ef8d2b620d50c6f6ae105e8f3ff9a4ba34be9dcb81b4f5bb0bc7: Status 404 returned error can't find the container with id 0f09906fcdd1ef8d2b620d50c6f6ae105e8f3ff9a4ba34be9dcb81b4f5bb0bc7 Oct 30 22:02:24.605391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.605368 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x"] Oct 30 22:02:24.606858 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.606827 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.606923 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.606907 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.606959 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.606937 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.659679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.659653 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb"] Oct 30 22:02:24.666672 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:02:24.666411 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26975667_9851_4f05_a4ff_a9ea7e08c769.slice/crio-eea01da825f7f71cd14152a38a60fb6d411c67cb1915c797f171cd6289345fd4 WatchSource:0}: Error finding container eea01da825f7f71cd14152a38a60fb6d411c67cb1915c797f171cd6289345fd4: Status 404 returned error can't find the container with id eea01da825f7f71cd14152a38a60fb6d411c67cb1915c797f171cd6289345fd4 Oct 30 22:02:24.668887 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.668852 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.668961 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.668937 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:24.669167 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:24.668974 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:02:25.011742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.011651 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" event={"ID":"adcaa605-fa45-4f88-ac6f-12a4d9b4363b","Type":"ContainerStarted","Data":"962fb143d1a66dea0b8186d64a586ed7c131ac6d6dc264af48bbb0b52894ce12"} Oct 30 22:02:25.012216 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.011743 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" event={"ID":"adcaa605-fa45-4f88-ac6f-12a4d9b4363b","Type":"ContainerStarted","Data":"0f09906fcdd1ef8d2b620d50c6f6ae105e8f3ff9a4ba34be9dcb81b4f5bb0bc7"} Oct 30 22:02:25.013813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.013787 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" event={"ID":"26975667-9851-4f05-a4ff-a9ea7e08c769","Type":"ContainerStarted","Data":"0f1c6ba2ec315b6ba3c265490bf523b2abf26065c6409f87e16b0b7854e12484"} Oct 30 22:02:25.013960 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.013828 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" event={"ID":"26975667-9851-4f05-a4ff-a9ea7e08c769","Type":"ContainerStarted","Data":"eea01da825f7f71cd14152a38a60fb6d411c67cb1915c797f171cd6289345fd4"} Oct 30 22:02:25.133662 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.133613 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podStartSLOduration=2.133593152 podStartE2EDuration="2.133593152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:25.130869115 +0000 UTC m=+1193.812019007" watchObservedRunningTime="2026-10-30 22:02:25.133593152 +0000 UTC m=+1193.814743047" Oct 30 22:02:25.173523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.172949 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:25.174369 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.174343 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:02:25.174512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.174389 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:02:25.222436 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.222407 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:25.223682 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.223659 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:02:25.223779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.223712 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 22:02:25.256833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:25.256741 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podStartSLOduration=2.256723599 podStartE2EDuration="2.256723599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:25.248176887 +0000 UTC m=+1193.929326817" watchObservedRunningTime="2026-10-30 22:02:25.256723599 +0000 UTC m=+1193.937873491" Oct 30 22:02:26.176516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:26.175881 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:02:26.176516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:26.175960 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:02:26.223096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:26.223066 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:02:26.223264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:26.223114 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 22:02:27.178524 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:27.177958 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:02:27.178524 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:27.178029 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:02:27.222839 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:27.222812 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:02:27.223004 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:27.222863 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 22:02:28.177904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:28.177868 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:02:28.178190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:28.178162 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:02:28.190688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:28.190649 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:02:28.223583 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:28.223553 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:02:28.223730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:28.223609 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 22:02:28.591516 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:02:28.591445 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:02:29.180516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:29.176777 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.180516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:29.176838 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:02:29.222741 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:29.222712 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:02:29.223171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:29.222760 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 22:02:30.178329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:30.178296 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:02:30.178664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:30.178634 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:02:30.223388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:30.223349 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:02:30.223819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:30.223402 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 22:02:31.172935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:31.172900 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:02:31.173125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:31.172972 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 22:02:31.223324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:31.223293 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:02:31.223437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:31.223346 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 22:02:31.686696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:31.686670 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:02:31.695570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:31.695549 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:02:32.429916 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:32.429887 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:32.430559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:32.430476 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:32.431379 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:32.431358 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:02:32.465840 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:32.465817 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:32.466084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:32.466067 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:32.466770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:32.466754 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:02:33.266506 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.266461 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c"] Oct 30 22:02:33.271474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.271456 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.324696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.324477 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:33.324696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.324590 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:33.324696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.324669 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:33.329805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.329778 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c"] Oct 30 22:02:33.371657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.371627 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:02:33.437222 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.437152 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.437603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.437293 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwzv\" (UniqueName: \"kubernetes.io/projected/b13960e0-0572-4d40-8c2e-41f6bfd5b877-kube-api-access-qgwzv\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.437603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.437345 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.538478 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.538450 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwzv\" (UniqueName: \"kubernetes.io/projected/b13960e0-0572-4d40-8c2e-41f6bfd5b877-kube-api-access-qgwzv\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.538664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.538534 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.538664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.538640 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.539324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.539305 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.541228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.541205 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.596793 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.596766 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwzv\" (UniqueName: \"kubernetes.io/projected/b13960e0-0572-4d40-8c2e-41f6bfd5b877-kube-api-access-qgwzv\") pod \"maas-api-revoke-sub-04f24c0e-bl52c\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:33.884877 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:33.884846 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:34.278417 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:02:34.278385 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13960e0_0572_4d40_8c2e_41f6bfd5b877.slice/crio-f412a60ec7bba108e69f31fbd010c7d067d631f77fda145629f854b0710770f0 WatchSource:0}: Error finding container f412a60ec7bba108e69f31fbd010c7d067d631f77fda145629f854b0710770f0: Status 404 returned error can't find the container with id f412a60ec7bba108e69f31fbd010c7d067d631f77fda145629f854b0710770f0 Oct 30 22:02:34.282160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:34.282135 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c"] Oct 30 22:02:35.142625 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:35.142586 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" event={"ID":"b13960e0-0572-4d40-8c2e-41f6bfd5b877","Type":"ContainerStarted","Data":"8d74e64084769a122c1a564d09846ae3a4b7cf80489dd7fac399b792fef1fd09"} Oct 30 22:02:35.143038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:35.142656 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" event={"ID":"b13960e0-0572-4d40-8c2e-41f6bfd5b877","Type":"ContainerStarted","Data":"f412a60ec7bba108e69f31fbd010c7d067d631f77fda145629f854b0710770f0"} Oct 30 22:02:35.260137 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:35.260074 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" podStartSLOduration=2.260055573 podStartE2EDuration="2.260055573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:35.259174606 +0000 UTC m=+1203.940324499" watchObservedRunningTime="2026-10-30 22:02:35.260055573 +0000 UTC m=+1203.941205467" Oct 30 22:02:36.152303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:36.152270 4933 generic.go:356] "Generic (PLEG): container finished" podID="b13960e0-0572-4d40-8c2e-41f6bfd5b877" containerID="8d74e64084769a122c1a564d09846ae3a4b7cf80489dd7fac399b792fef1fd09" exitCode=0 Oct 30 22:02:36.152736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:36.152317 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" event={"ID":"b13960e0-0572-4d40-8c2e-41f6bfd5b877","Type":"ContainerDied","Data":"8d74e64084769a122c1a564d09846ae3a4b7cf80489dd7fac399b792fef1fd09"} Oct 30 22:02:37.315633 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.315611 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:37.398876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.398848 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-xks-ca\") pod \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " Oct 30 22:02:37.399028 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.398907 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-openshift-ca\") pod \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " Oct 30 22:02:37.399028 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.399015 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13960e0-0572-4d40-8c2e-41f6bfd5b877-kube-api-access-qgwzv\" (UniqueName: \"kubernetes.io/projected/b13960e0-0572-4d40-8c2e-41f6bfd5b877-kube-api-access-qgwzv\") pod \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\" (UID: \"b13960e0-0572-4d40-8c2e-41f6bfd5b877\") " Oct 30 22:02:37.399270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.399244 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-openshift-ca" pod "b13960e0-0572-4d40-8c2e-41f6bfd5b877" (UID: "b13960e0-0572-4d40-8c2e-41f6bfd5b877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:37.399464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.399442 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:37.401081 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.401063 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-xks-ca" pod "b13960e0-0572-4d40-8c2e-41f6bfd5b877" (UID: "b13960e0-0572-4d40-8c2e-41f6bfd5b877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:37.401189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.401129 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13960e0-0572-4d40-8c2e-41f6bfd5b877-kube-api-access-qgwzv" pod "b13960e0-0572-4d40-8c2e-41f6bfd5b877" (UID: "b13960e0-0572-4d40-8c2e-41f6bfd5b877"). InnerVolumeSpecName "kube-api-access-qgwzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:37.500597 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.500312 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgwzv\" (UniqueName: \"kubernetes.io/projected/b13960e0-0572-4d40-8c2e-41f6bfd5b877-kube-api-access-qgwzv\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:37.500597 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:37.500340 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b13960e0-0572-4d40-8c2e-41f6bfd5b877-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.179058 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:38.179027 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" Oct 30 22:02:38.179238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:38.179066 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c" event={"ID":"b13960e0-0572-4d40-8c2e-41f6bfd5b877","Type":"ContainerDied","Data":"f412a60ec7bba108e69f31fbd010c7d067d631f77fda145629f854b0710770f0"} Oct 30 22:02:38.179238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:38.179156 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f412a60ec7bba108e69f31fbd010c7d067d631f77fda145629f854b0710770f0" Oct 30 22:02:45.067578 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:45.065194 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx"] Oct 30 22:02:45.107464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:45.107428 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bc7fd-vtlsx"] Oct 30 22:02:45.506873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:45.506793 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724df9e7-a661-4d50-8929-8a1f6d499bf4" path="/var/lib/kubelet/pods/724df9e7-a661-4d50-8929-8a1f6d499bf4/volumes" Oct 30 22:02:47.353887 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.353851 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm"] Oct 30 22:02:47.354670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.354652 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b13960e0-0572-4d40-8c2e-41f6bfd5b877" containerName="revoke-keys" Oct 30 22:02:47.354772 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.354670 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13960e0-0572-4d40-8c2e-41f6bfd5b877" containerName="revoke-keys" Oct 30 22:02:47.354842 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.354830 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13960e0-0572-4d40-8c2e-41f6bfd5b877" containerName="revoke-keys" Oct 30 22:02:47.358766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.358745 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.412530 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.412481 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm"] Oct 30 22:02:47.438547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.438517 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvm7c\" (UniqueName: \"kubernetes.io/projected/c5363e34-af60-478e-ad07-07bb06305a80-kube-api-access-gvm7c\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.438708 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.438586 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-home\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.438708 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.438693 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5363e34-af60-478e-ad07-07bb06305a80-tls-certs\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.438824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.438731 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-tmp-dir\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.438963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.438936 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-dshm\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.439012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.438995 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-model-cache\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.457465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.457440 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d516c4\"/\"distinct-2-w1-d516c4-kserve-self-signed-certs\"" Oct 30 22:02:47.540740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.540700 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm7c\" (UniqueName: \"kubernetes.io/projected/c5363e34-af60-478e-ad07-07bb06305a80-kube-api-access-gvm7c\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.540932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.540793 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-home\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.540932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.540851 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5363e34-af60-478e-ad07-07bb06305a80-tls-certs\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.540932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.540891 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-tmp-dir\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.541054 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.541043 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-dshm\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.541107 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.541088 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-model-cache\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.541389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.541365 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-tmp-dir\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.541621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.541548 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-model-cache\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.541937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.541633 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-home\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.543606 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.543583 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-dshm\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.543776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.543754 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5363e34-af60-478e-ad07-07bb06305a80-tls-certs\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.599267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.599238 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm7c\" (UniqueName: \"kubernetes.io/projected/c5363e34-af60-478e-ad07-07bb06305a80-kube-api-access-gvm7c\") pod \"distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:47.675079 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:47.674994 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:48.174584 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:02:48.174550 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5363e34_af60_478e_ad07_07bb06305a80.slice/crio-16a714b778569022d38179403abbdde88ccb624dcc71203e5c1f226f95c6fedd WatchSource:0}: Error finding container 16a714b778569022d38179403abbdde88ccb624dcc71203e5c1f226f95c6fedd: Status 404 returned error can't find the container with id 16a714b778569022d38179403abbdde88ccb624dcc71203e5c1f226f95c6fedd Oct 30 22:02:48.175311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:48.175288 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm"] Oct 30 22:02:48.292038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:48.292003 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" event={"ID":"c5363e34-af60-478e-ad07-07bb06305a80","Type":"ContainerStarted","Data":"825b22aad78ea35706bf0de3d4d1c7f213a0d19457da0951bde82b8ceb985569"} Oct 30 22:02:48.292186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:48.292054 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" event={"ID":"c5363e34-af60-478e-ad07-07bb06305a80","Type":"ContainerStarted","Data":"16a714b778569022d38179403abbdde88ccb624dcc71203e5c1f226f95c6fedd"} Oct 30 22:02:48.407844 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:48.407791 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podStartSLOduration=1.407776943 podStartE2EDuration="1.407776943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:48.406152982 +0000 UTC m=+1217.087302876" watchObservedRunningTime="2026-10-30 22:02:48.407776943 +0000 UTC m=+1217.088926836" Oct 30 22:02:48.675823 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:48.675794 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:49.038213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.038172 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb"] Oct 30 22:02:49.042334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.042309 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.060658 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.060631 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t"] Oct 30 22:02:49.088874 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.088848 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04e8d550-tdz7t"] Oct 30 22:02:49.104281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.104258 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:02:49.104411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.104303 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:02:49.109217 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.109195 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb"] Oct 30 22:02:49.135087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.135055 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:02:49.135212 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.135132 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:02:49.176764 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.176737 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.176921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.176804 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.176985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.176946 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sl45\" (UniqueName: \"kubernetes.io/projected/53945f04-2719-4de3-872e-b06d21ee17ad-kube-api-access-6sl45\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.277970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.277922 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.278181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.278039 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sl45\" (UniqueName: \"kubernetes.io/projected/53945f04-2719-4de3-872e-b06d21ee17ad-kube-api-access-6sl45\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.278181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.278123 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.278830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.278803 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.280665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.280645 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.335993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.335929 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sl45\" (UniqueName: \"kubernetes.io/projected/53945f04-2719-4de3-872e-b06d21ee17ad-kube-api-access-6sl45\") pod \"maas-api-revoke-sub-b7de04a9-mfqgb\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.357520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.357472 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:49.477512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.474680 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06005574-ed44-4fec-83bd-7964ebb58b66" path="/var/lib/kubelet/pods/06005574-ed44-4fec-83bd-7964ebb58b66/volumes" Oct 30 22:02:49.734378 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:02:49.734337 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53945f04_2719_4de3_872e_b06d21ee17ad.slice/crio-80b2e6b9738bdcc9b5438d325f950abcfebc12c0b6db1e48335969d78fa0f388 WatchSource:0}: Error finding container 80b2e6b9738bdcc9b5438d325f950abcfebc12c0b6db1e48335969d78fa0f388: Status 404 returned error can't find the container with id 80b2e6b9738bdcc9b5438d325f950abcfebc12c0b6db1e48335969d78fa0f388 Oct 30 22:02:49.737385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:49.737358 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb"] Oct 30 22:02:50.308867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:50.308775 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" event={"ID":"53945f04-2719-4de3-872e-b06d21ee17ad","Type":"ContainerStarted","Data":"15f01a0e90c8a053db00d800279d8d981eb378d68b238a09a8f287015bbc52fa"} Oct 30 22:02:50.309057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:50.308863 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" event={"ID":"53945f04-2719-4de3-872e-b06d21ee17ad","Type":"ContainerStarted","Data":"80b2e6b9738bdcc9b5438d325f950abcfebc12c0b6db1e48335969d78fa0f388"} Oct 30 22:02:50.427214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:50.427157 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" podStartSLOduration=2.427142476 podStartE2EDuration="2.427142476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:50.423616064 +0000 UTC m=+1219.104765960" watchObservedRunningTime="2026-10-30 22:02:50.427142476 +0000 UTC m=+1219.108292368" Oct 30 22:02:51.318503 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:51.318466 4933 generic.go:356] "Generic (PLEG): container finished" podID="53945f04-2719-4de3-872e-b06d21ee17ad" containerID="15f01a0e90c8a053db00d800279d8d981eb378d68b238a09a8f287015bbc52fa" exitCode=0 Oct 30 22:02:51.318886 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:51.318552 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" event={"ID":"53945f04-2719-4de3-872e-b06d21ee17ad","Type":"ContainerDied","Data":"15f01a0e90c8a053db00d800279d8d981eb378d68b238a09a8f287015bbc52fa"} Oct 30 22:02:52.495062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.495034 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:52.619323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.619243 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53945f04-2719-4de3-872e-b06d21ee17ad-kube-api-access-6sl45\" (UniqueName: \"kubernetes.io/projected/53945f04-2719-4de3-872e-b06d21ee17ad-kube-api-access-6sl45\") pod \"53945f04-2719-4de3-872e-b06d21ee17ad\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " Oct 30 22:02:52.619323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.619290 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-xks-ca\") pod \"53945f04-2719-4de3-872e-b06d21ee17ad\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " Oct 30 22:02:52.619556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.619435 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-openshift-ca\") pod \"53945f04-2719-4de3-872e-b06d21ee17ad\" (UID: \"53945f04-2719-4de3-872e-b06d21ee17ad\") " Oct 30 22:02:52.623701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.623664 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-openshift-ca" pod "53945f04-2719-4de3-872e-b06d21ee17ad" (UID: "53945f04-2719-4de3-872e-b06d21ee17ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:52.627728 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.627695 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-xks-ca" pod "53945f04-2719-4de3-872e-b06d21ee17ad" (UID: "53945f04-2719-4de3-872e-b06d21ee17ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:52.627844 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.627727 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53945f04-2719-4de3-872e-b06d21ee17ad-kube-api-access-6sl45" pod "53945f04-2719-4de3-872e-b06d21ee17ad" (UID: "53945f04-2719-4de3-872e-b06d21ee17ad"). InnerVolumeSpecName "kube-api-access-6sl45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:52.721026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.720999 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sl45\" (UniqueName: \"kubernetes.io/projected/53945f04-2719-4de3-872e-b06d21ee17ad-kube-api-access-6sl45\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:52.721026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.721028 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:52.721196 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:52.721043 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53945f04-2719-4de3-872e-b06d21ee17ad-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:02:53.340147 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:53.340113 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" Oct 30 22:02:53.340341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:53.340156 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb" event={"ID":"53945f04-2719-4de3-872e-b06d21ee17ad","Type":"ContainerDied","Data":"80b2e6b9738bdcc9b5438d325f950abcfebc12c0b6db1e48335969d78fa0f388"} Oct 30 22:02:53.340341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:53.340211 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b2e6b9738bdcc9b5438d325f950abcfebc12c0b6db1e48335969d78fa0f388" Oct 30 22:02:57.675378 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:57.675346 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:57.689289 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:57.689265 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:58.395545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:02:58.395481 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:02:58.554758 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:02:58.554720 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:03:00.050257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:00.050224 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs"] Oct 30 22:03:00.098758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:00.098730 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e490d292-4blrs"] Oct 30 22:03:01.466717 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:01.466688 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428266de-5106-4b55-bbe1-64e9ee734f75" path="/var/lib/kubelet/pods/428266de-5106-4b55-bbe1-64e9ee734f75/volumes" Oct 30 22:03:05.052010 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:05.051978 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6"] Oct 30 22:03:05.098951 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:05.098917 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e44eff9-d7cw6"] Oct 30 22:03:05.470556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:05.470452 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a8239a-4975-4b5b-92aa-7b00144c9bad" path="/var/lib/kubelet/pods/e7a8239a-4975-4b5b-92aa-7b00144c9bad/volumes" Oct 30 22:03:09.424061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.424025 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td"] Oct 30 22:03:09.424582 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.424569 4933 cpu_manager.go:414] "containerMap: removing container" podUID="53945f04-2719-4de3-872e-b06d21ee17ad" containerName="revoke-keys" Oct 30 22:03:09.424636 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.424583 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53945f04-2719-4de3-872e-b06d21ee17ad" containerName="revoke-keys" Oct 30 22:03:09.424708 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.424699 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="53945f04-2719-4de3-872e-b06d21ee17ad" containerName="revoke-keys" Oct 30 22:03:09.429559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.429537 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.477329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.477304 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs"] Oct 30 22:03:09.482083 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.482064 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.528313 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.528292 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td"] Oct 30 22:03:09.530659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.530637 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854xj\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-kube-api-access-854xj\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.530787 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.530705 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.530848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.530781 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.530848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.530805 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.530940 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.530917 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.531034 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531018 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.531072 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531047 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.531111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531077 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgj62\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-kube-api-access-bgj62\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.531152 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531118 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.531152 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531144 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.531258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531160 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.531316 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531266 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.531316 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531298 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.531431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531413 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.531469 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531444 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.531531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531482 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.531588 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531572 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.531632 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.531597 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.588474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.588405 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs"] Oct 30 22:03:09.633202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633174 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633243 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633258 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633284 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgj62\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-kube-api-access-bgj62\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633305 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633330 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633356 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633404 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633426 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633482 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633530 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633570 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633650 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633689 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633765 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854xj\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-kube-api-access-854xj\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633875 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633888 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.633976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633927 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.633976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633952 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.634302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.634280 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.635731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.633647 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.635731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.634942 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.635731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.635083 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-credential-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.635731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.635370 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.635731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.635589 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istiod-ca-cert\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.636080 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.636039 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-certs\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.636612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.636553 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.636855 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.636807 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-data\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.637110 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.637084 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-socket\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.638024 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.637995 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.639222 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.639198 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-envoy\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.639842 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.639799 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-podinfo\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.693021 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.691990 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.693021 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.692199 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-token\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.694542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.693833 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854xj\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-kube-api-access-854xj\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:09.696523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.696470 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgj62\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-kube-api-access-bgj62\") pod \"e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.745223 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.745194 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:09.800329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:09.798683 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:10.192324 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:03:10.192296 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc883ffae_9b9a_4622_9199_9aebffa4a5e6.slice/crio-2c569cb956fc7548224c8e8f8719c53da2fa9fde34464260264954e2f024dc22 WatchSource:0}: Error finding container 2c569cb956fc7548224c8e8f8719c53da2fa9fde34464260264954e2f024dc22: Status 404 returned error can't find the container with id 2c569cb956fc7548224c8e8f8719c53da2fa9fde34464260264954e2f024dc22 Oct 30 22:03:10.193650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.193629 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td"] Oct 30 22:03:10.194824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.194789 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.194928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.194865 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.194928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.194913 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.251138 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:03:10.251110 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7443bc_8680_45f0_b375_2d1c7a8d6dac.slice/crio-0001f93fa050775210d0c604fffab7004983530c4ad2d7fd484a57bc6588d1be WatchSource:0}: Error finding container 0001f93fa050775210d0c604fffab7004983530c4ad2d7fd484a57bc6588d1be: Status 404 returned error can't find the container with id 0001f93fa050775210d0c604fffab7004983530c4ad2d7fd484a57bc6588d1be Oct 30 22:03:10.253554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.253420 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs"] Oct 30 22:03:10.254005 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.253964 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.254099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.254048 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.254099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.254088 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:10.521700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.521571 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" event={"ID":"fa7443bc-8680-45f0-b375-2d1c7a8d6dac","Type":"ContainerStarted","Data":"55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34"} Oct 30 22:03:10.521700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.521686 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" event={"ID":"fa7443bc-8680-45f0-b375-2d1c7a8d6dac","Type":"ContainerStarted","Data":"0001f93fa050775210d0c604fffab7004983530c4ad2d7fd484a57bc6588d1be"} Oct 30 22:03:10.532400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.532310 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" event={"ID":"c883ffae-9b9a-4622-9199-9aebffa4a5e6","Type":"ContainerStarted","Data":"2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c"} Oct 30 22:03:10.532400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.532390 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" event={"ID":"c883ffae-9b9a-4622-9199-9aebffa4a5e6","Type":"ContainerStarted","Data":"2c569cb956fc7548224c8e8f8719c53da2fa9fde34464260264954e2f024dc22"} Oct 30 22:03:10.677514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.673866 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podStartSLOduration=1.673847679 podStartE2EDuration="1.673847679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:10.667445334 +0000 UTC m=+1239.348595229" watchObservedRunningTime="2026-10-30 22:03:10.673847679 +0000 UTC m=+1239.354997573" Oct 30 22:03:10.746513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.746133 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:10.796312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.796223 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podStartSLOduration=1.796204709 podStartE2EDuration="1.796204709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:10.792904563 +0000 UTC m=+1239.474054458" watchObservedRunningTime="2026-10-30 22:03:10.796204709 +0000 UTC m=+1239.477354599" Oct 30 22:03:10.800121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:10.800090 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:11.754515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:11.754363 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:11.754515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:11.754429 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:11.801775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:11.801738 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:11.801921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:11.801789 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:12.748256 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:12.747833 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:12.748256 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:12.747894 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:12.800659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:12.800626 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:12.801098 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:12.800676 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:13.751517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:13.748827 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:13.751517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:13.748902 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:13.800238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:13.800192 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:13.800419 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:13.800269 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:13.973518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:13.973286 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:03:13.997049 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:13.997010 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq"] Oct 30 22:03:14.002135 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.002068 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.062512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.061818 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:14.062718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.062565 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:03:14.062718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.062641 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:14.062833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.062733 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:14.089589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.087197 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq"] Oct 30 22:03:14.114927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.114673 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.114927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.114804 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frsm\" (UniqueName: \"kubernetes.io/projected/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-kube-api-access-2frsm\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.115398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.115169 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.216562 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.216524 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.216747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.216599 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.216747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.216627 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frsm\" (UniqueName: \"kubernetes.io/projected/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-kube-api-access-2frsm\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.217572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.217507 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.219281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.219250 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.292873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.292833 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frsm\" (UniqueName: \"kubernetes.io/projected/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-kube-api-access-2frsm\") pod \"maas-api-revoke-sub-014b72ba-mnbkq\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.320224 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.319944 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:14.842513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.835192 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:14.842513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.835256 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:14.925511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.922895 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:14.925511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:14.922945 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:15.027518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:15.014592 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq"] Oct 30 22:03:15.749513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:15.748884 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:15.749513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:15.748944 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:15.820516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:15.819597 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:15.820516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:15.819676 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:16.065518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.063964 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" event={"ID":"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a","Type":"ContainerStarted","Data":"74c1e0b472508bad52c67f35d11c9e18df662aa67b4f60b81dda19008293eab0"} Oct 30 22:03:16.065518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.064037 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" event={"ID":"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a","Type":"ContainerStarted","Data":"9716d44a3793762d16ba8e575c5cdd5fe0cc960e3d32fd23764e10a9adead332"} Oct 30 22:03:16.388752 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.388647 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.490516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.489582 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.726692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.726587 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.728091 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.726903 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:03:16.728734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.726940 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.758826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.758795 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:16.759047 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.759020 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:16.817006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.816975 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:16.817268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.817224 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:16.843641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.843570 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.907519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:16.905570 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:03:17.103567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.068548 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.325516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.316582 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.360735 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.360608 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:03:17.410519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.410357 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:03:17.428181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.428133 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.428868 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.428845 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:03:17.513351 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.513311 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:03:17.555517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.515670 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:03:17.556078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.532740 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.556252 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.540909 4933 generic.go:356] "Generic (PLEG): container finished" podID="5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" containerID="74c1e0b472508bad52c67f35d11c9e18df662aa67b4f60b81dda19008293eab0" exitCode=0 Oct 30 22:03:17.598510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.598452 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.637261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.637177 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:03:17.679667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.679624 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" event={"ID":"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a","Type":"ContainerDied","Data":"74c1e0b472508bad52c67f35d11c9e18df662aa67b4f60b81dda19008293eab0"} Oct 30 22:03:17.710517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.697798 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.742961 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.742914 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.850647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.746804 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.851625 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.851602 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:03:17.852064 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.852048 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:03:17.879971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.771074 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:17.881166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.881129 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:17.881379 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.881351 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:17.905725 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.905638 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:17.905942 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.905918 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:17.989553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.923439 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.073406 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.923741 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:03:18.098642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.932894 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:03:18.104529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:17.933592 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.104529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.005839 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:03:18.113537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.109010 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:03:18.113537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.109101 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:03:18.113537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.078590 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.113858 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.084361 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" podStartSLOduration=5.084335179 podStartE2EDuration="5.084335179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:16.199795026 +0000 UTC m=+1244.880944921" watchObservedRunningTime="2026-10-30 22:03:18.084335179 +0000 UTC m=+1246.765485073" Oct 30 22:03:18.118451 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.118427 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:03:18.172555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.172465 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:03:18.372832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.372791 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.384216 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.384175 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:03:18.413333 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.413295 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.413702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.413597 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.426648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.426577 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:03:18.426841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.426826 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:03:18.426983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.426970 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:03:18.565345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.564159 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:03:18.600463 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.577014 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:03:18.600463 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.578520 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:03:18.608514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.600859 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:03:18.631513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.608174 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:18.632437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.632413 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:03:18.681447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.681368 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:03:18.687999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.687979 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:03:18.703156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.699932 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:03:18.778528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.762906 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:03:18.778528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.767229 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 30 22:03:18.778528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.767276 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 30 22:03:18.837656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.816559 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": context deadline exceeded" Oct 30 22:03:18.837656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.816939 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:03:18.837656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.816981 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 22:03:18.842729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.842596 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:03:18.926135 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:18.925551 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:03:19.011558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.003830 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:03:19.133958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.116618 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:03:19.157074 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.136636 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:03:19.189411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.187629 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:19.304514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.292653 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:19.323690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.305533 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-kube-api-access-2frsm\" (UniqueName: \"kubernetes.io/projected/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-kube-api-access-2frsm\") pod \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " Oct 30 22:03:19.323690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.317267 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-xks-ca\") pod \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " Oct 30 22:03:19.323690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.317476 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-openshift-ca\") pod \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\" (UID: \"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a\") " Oct 30 22:03:19.323690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.308304 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:03:19.323690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.319822 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:03:19.323690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.321860 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-openshift-ca" pod "5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" (UID: "5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:19.344521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.342069 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-xks-ca" pod "5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" (UID: "5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:19.344797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.344770 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-kube-api-access-2frsm" pod "5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" (UID: "5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a"). InnerVolumeSpecName "kube-api-access-2frsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:19.420996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.420963 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:19.421188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.421011 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2frsm\" (UniqueName: \"kubernetes.io/projected/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-kube-api-access-2frsm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:19.421188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.421028 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:19.933512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.933342 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" event={"ID":"5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a","Type":"ContainerDied","Data":"9716d44a3793762d16ba8e575c5cdd5fe0cc960e3d32fd23764e10a9adead332"} Oct 30 22:03:19.933512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.933482 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq" Oct 30 22:03:19.933512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.933517 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9716d44a3793762d16ba8e575c5cdd5fe0cc960e3d32fd23764e10a9adead332" Oct 30 22:03:19.951896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.951853 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:19.953848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.953820 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:19.956874 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:19.956857 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:03:20.292263 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:20.292223 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:20.293665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:20.293642 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:20.297943 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:20.297923 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:03:21.055515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:21.055007 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp"] Oct 30 22:03:21.083615 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:21.064756 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1013e658-dsmxp"] Oct 30 22:03:21.527425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:21.527395 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb4666e-865e-4c47-8b78-6b13f7601b8d" path="/var/lib/kubelet/pods/aeb4666e-865e-4c47-8b78-6b13f7601b8d/volumes" Oct 30 22:03:22.054540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:22.053331 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm"] Oct 30 22:03:22.062080 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:22.062049 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-548685a1-s5thm"] Oct 30 22:03:23.469993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:23.469960 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55152a9c-b3db-4e2a-a309-c493e0d84ad8" path="/var/lib/kubelet/pods/55152a9c-b3db-4e2a-a309-c493e0d84ad8/volumes" Oct 30 22:03:28.546083 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:03:28.545997 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:03:29.342900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.342868 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d"] Oct 30 22:03:29.343433 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.343420 4933 cpu_manager.go:414] "containerMap: removing container" podUID="5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" containerName="revoke-keys" Oct 30 22:03:29.343536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.343437 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" containerName="revoke-keys" Oct 30 22:03:29.343583 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.343553 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" containerName="revoke-keys" Oct 30 22:03:29.348539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.348483 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.388361 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.388334 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbd6c\" (UniqueName: \"kubernetes.io/projected/104c12f7-935c-41e5-b8e2-8844f45a4ace-kube-api-access-gbd6c\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.388527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.388484 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.388581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.388545 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.404411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.404386 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d"] Oct 30 22:03:29.449790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.449766 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:29.449937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.449840 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:29.450013 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.449954 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:03:29.450064 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.450019 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:29.490250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.490227 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.490371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.490264 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.490452 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.490433 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbd6c\" (UniqueName: \"kubernetes.io/projected/104c12f7-935c-41e5-b8e2-8844f45a4ace-kube-api-access-gbd6c\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.490929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.490910 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.492901 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.492877 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.548657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.548631 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbd6c\" (UniqueName: \"kubernetes.io/projected/104c12f7-935c-41e5-b8e2-8844f45a4ace-kube-api-access-gbd6c\") pod \"maas-api-revoke-sub-6ce908ea-glg6d\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:29.662276 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:29.662217 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:30.057669 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:03:30.057638 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104c12f7_935c_41e5_b8e2_8844f45a4ace.slice/crio-d61d6d02738276b45f4072790c81107201f11ef3cc9bd11a249db9eae693fa9a WatchSource:0}: Error finding container d61d6d02738276b45f4072790c81107201f11ef3cc9bd11a249db9eae693fa9a: Status 404 returned error can't find the container with id d61d6d02738276b45f4072790c81107201f11ef3cc9bd11a249db9eae693fa9a Oct 30 22:03:30.057862 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:30.057838 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d"] Oct 30 22:03:30.089714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:30.089685 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" event={"ID":"104c12f7-935c-41e5-b8e2-8844f45a4ace","Type":"ContainerStarted","Data":"d61d6d02738276b45f4072790c81107201f11ef3cc9bd11a249db9eae693fa9a"} Oct 30 22:03:31.048241 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:31.048214 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2"] Oct 30 22:03:31.099631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:31.099602 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2304ee4f-grhx2"] Oct 30 22:03:31.099861 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:31.099840 4933 generic.go:356] "Generic (PLEG): container finished" podID="104c12f7-935c-41e5-b8e2-8844f45a4ace" containerID="cc7b4656a2a0c3df484f25ea0977ef754dae61e582dcfc48d4818bd3ca36a569" exitCode=0 Oct 30 22:03:31.099962 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:31.099922 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" event={"ID":"104c12f7-935c-41e5-b8e2-8844f45a4ace","Type":"ContainerDied","Data":"cc7b4656a2a0c3df484f25ea0977ef754dae61e582dcfc48d4818bd3ca36a569"} Oct 30 22:03:31.468272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:31.468240 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01f96a5-59c1-4e35-8c31-d4a01e190821" path="/var/lib/kubelet/pods/d01f96a5-59c1-4e35-8c31-d4a01e190821/volumes" Oct 30 22:03:31.707523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:31.707379 4933 scope.go:122] "RemoveContainer" containerID="dfbc4b424cad3af5d43f8ce0f20b0d3a7cef13900f5de2da77ccfe6817b31177" Oct 30 22:03:32.277347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.277322 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:32.325789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.325758 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-xks-ca\") pod \"104c12f7-935c-41e5-b8e2-8844f45a4ace\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " Oct 30 22:03:32.325942 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.325925 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-openshift-ca\") pod \"104c12f7-935c-41e5-b8e2-8844f45a4ace\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " Oct 30 22:03:32.326074 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.326062 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104c12f7-935c-41e5-b8e2-8844f45a4ace-kube-api-access-gbd6c\" (UniqueName: \"kubernetes.io/projected/104c12f7-935c-41e5-b8e2-8844f45a4ace-kube-api-access-gbd6c\") pod \"104c12f7-935c-41e5-b8e2-8844f45a4ace\" (UID: \"104c12f7-935c-41e5-b8e2-8844f45a4ace\") " Oct 30 22:03:32.326246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.326225 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-openshift-ca" pod "104c12f7-935c-41e5-b8e2-8844f45a4ace" (UID: "104c12f7-935c-41e5-b8e2-8844f45a4ace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:32.326521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.326505 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:32.327904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.327879 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-xks-ca" pod "104c12f7-935c-41e5-b8e2-8844f45a4ace" (UID: "104c12f7-935c-41e5-b8e2-8844f45a4ace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:32.328413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.328396 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104c12f7-935c-41e5-b8e2-8844f45a4ace-kube-api-access-gbd6c" pod "104c12f7-935c-41e5-b8e2-8844f45a4ace" (UID: "104c12f7-935c-41e5-b8e2-8844f45a4ace"). InnerVolumeSpecName "kube-api-access-gbd6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:32.428134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.428054 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104c12f7-935c-41e5-b8e2-8844f45a4ace-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:32.428134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:32.428085 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbd6c\" (UniqueName: \"kubernetes.io/projected/104c12f7-935c-41e5-b8e2-8844f45a4ace-kube-api-access-gbd6c\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:33.117797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:33.117741 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" Oct 30 22:03:33.117797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:33.117771 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d" event={"ID":"104c12f7-935c-41e5-b8e2-8844f45a4ace","Type":"ContainerDied","Data":"d61d6d02738276b45f4072790c81107201f11ef3cc9bd11a249db9eae693fa9a"} Oct 30 22:03:33.118055 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:33.117836 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d61d6d02738276b45f4072790c81107201f11ef3cc9bd11a249db9eae693fa9a" Oct 30 22:03:40.053027 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.052996 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv"] Oct 30 22:03:40.053818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.053796 4933 cpu_manager.go:414] "containerMap: removing container" podUID="104c12f7-935c-41e5-b8e2-8844f45a4ace" containerName="revoke-keys" Oct 30 22:03:40.053924 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.053820 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104c12f7-935c-41e5-b8e2-8844f45a4ace" containerName="revoke-keys" Oct 30 22:03:40.054033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.054019 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="104c12f7-935c-41e5-b8e2-8844f45a4ace" containerName="revoke-keys" Oct 30 22:03:40.057594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.057573 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.225482 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.225441 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.225669 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.225600 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.225713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.225669 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srl4n\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-kube-api-access-srl4n\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.225807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.225786 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.225847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.225828 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.225892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.225852 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.225995 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.225975 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e722ade8-3623-4a25-af85-19ba70f18ab0-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.226119 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.226101 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.226195 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.226180 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.280989 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.280958 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv"] Oct 30 22:03:40.327873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.327800 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.327873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.327848 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.327891 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.327932 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srl4n\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-kube-api-access-srl4n\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.327967 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328006 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328039 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328084 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e722ade8-3623-4a25-af85-19ba70f18ab0-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328173 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328386 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-certs\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328535 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328475 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328630 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-data\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328631 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-credential-socket\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.328921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.328903 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e722ade8-3623-4a25-af85-19ba70f18ab0-istiod-ca-cert\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.330373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.330343 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-envoy\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.330621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.330599 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-podinfo\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.389998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.389972 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-token\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.392707 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.392679 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srl4n\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-kube-api-access-srl4n\") pod \"e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.402272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.402252 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff"] Oct 30 22:03:40.415156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.415131 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.466793 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.466753 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff"] Oct 30 22:03:40.531039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.530995 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531044 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531193 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531220 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531261 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531382 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531361 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531430 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531391 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531518 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.531613 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.531575 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-kube-api-access-nqdnq\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632560 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632591 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632637 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632652 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632677 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632708 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632730 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632781 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.632833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.632823 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-kube-api-access-nqdnq\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.633248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.633063 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.633348 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.633324 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-certs\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.633408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.633356 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-credential-socket\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.633408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.633396 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-data\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.633686 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.633670 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istiod-ca-cert\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.635245 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.635222 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-envoy\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.635591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.635573 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-podinfo\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.671583 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.671562 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:40.691164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.691136 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-token\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.691453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.691432 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-kube-api-access-nqdnq\") pod \"e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:40.733947 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:40.732867 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:41.090583 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.090551 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq"] Oct 30 22:03:41.102777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.102749 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91e7cec-csshq"] Oct 30 22:03:41.140246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.140223 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv"] Oct 30 22:03:41.145443 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:03:41.145420 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode722ade8_3623_4a25_af85_19ba70f18ab0.slice/crio-a3fe4106fb33a380f6a04a1c8fa96ba46cd07f93b246944f113b1aa523e0a985 WatchSource:0}: Error finding container a3fe4106fb33a380f6a04a1c8fa96ba46cd07f93b246944f113b1aa523e0a985: Status 404 returned error can't find the container with id a3fe4106fb33a380f6a04a1c8fa96ba46cd07f93b246944f113b1aa523e0a985 Oct 30 22:03:41.148276 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.148237 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.148364 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.148322 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.148408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.148372 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.192907 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.192864 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" event={"ID":"e722ade8-3623-4a25-af85-19ba70f18ab0","Type":"ContainerStarted","Data":"a3fe4106fb33a380f6a04a1c8fa96ba46cd07f93b246944f113b1aa523e0a985"} Oct 30 22:03:41.211948 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:03:41.211921 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488cb661_6f7f_4ba3_b7d2_826dc69fcd94.slice/crio-2d4738b0f441424af7c1c27e3738e90a9362f7b4e2c765ee14f1a534c4abcdeb WatchSource:0}: Error finding container 2d4738b0f441424af7c1c27e3738e90a9362f7b4e2c765ee14f1a534c4abcdeb: Status 404 returned error can't find the container with id 2d4738b0f441424af7c1c27e3738e90a9362f7b4e2c765ee14f1a534c4abcdeb Oct 30 22:03:41.214019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.214001 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff"] Oct 30 22:03:41.214835 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.214808 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.214899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.214863 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.214899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.214891 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:03:41.473285 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:41.473198 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735d1747-398e-4fee-a712-24a4a4b4a7b1" path="/var/lib/kubelet/pods/735d1747-398e-4fee-a712-24a4a4b4a7b1/volumes" Oct 30 22:03:42.205860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.205822 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" event={"ID":"488cb661-6f7f-4ba3-b7d2-826dc69fcd94","Type":"ContainerStarted","Data":"14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1"} Oct 30 22:03:42.206343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.205887 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" event={"ID":"488cb661-6f7f-4ba3-b7d2-826dc69fcd94","Type":"ContainerStarted","Data":"2d4738b0f441424af7c1c27e3738e90a9362f7b4e2c765ee14f1a534c4abcdeb"} Oct 30 22:03:42.208481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.208458 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" event={"ID":"e722ade8-3623-4a25-af85-19ba70f18ab0","Type":"ContainerStarted","Data":"8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225"} Oct 30 22:03:42.562548 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.562124 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podStartSLOduration=2.562100119 podStartE2EDuration="2.562100119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:42.428164895 +0000 UTC m=+1271.109314787" watchObservedRunningTime="2026-10-30 22:03:42.562100119 +0000 UTC m=+1271.243250015" Oct 30 22:03:42.603033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.602960 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podStartSLOduration=2.602939982 podStartE2EDuration="2.602939982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:42.564854841 +0000 UTC m=+1271.246004736" watchObservedRunningTime="2026-10-30 22:03:42.602939982 +0000 UTC m=+1271.284089875" Oct 30 22:03:42.675513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.674827 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:42.679510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.678597 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:03:42.679510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.678653 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:03:42.742517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.739435 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:42.742517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.739593 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:03:42.742517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:42.739637 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 22:03:43.598367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.598332 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn"] Oct 30 22:03:43.602901 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.602882 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.663527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.662775 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:43.687526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.676818 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:03:43.687526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.676871 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:03:43.701525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.697762 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn"] Oct 30 22:03:43.715512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.714280 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqv59\" (UniqueName: \"kubernetes.io/projected/3b7d5514-83c1-488f-813a-542407800d2a-kube-api-access-pqv59\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.715512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.714329 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.715512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.714382 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.715512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.714801 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:43.715512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.714873 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:03:43.715512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.714991 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:43.798553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.796013 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:03:43.798553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.796066 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 22:03:43.815209 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.815172 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.815434 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.815411 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqv59\" (UniqueName: \"kubernetes.io/projected/3b7d5514-83c1-488f-813a-542407800d2a-kube-api-access-pqv59\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.815477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.815447 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.816238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.816212 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.817993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.817949 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.897310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.897151 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqv59\" (UniqueName: \"kubernetes.io/projected/3b7d5514-83c1-488f-813a-542407800d2a-kube-api-access-pqv59\") pod \"maas-api-revoke-sub-25222a92-pn7xn\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:43.919034 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:43.919009 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:44.495859 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:44.495764 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn"] Oct 30 22:03:44.673581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:44.672868 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:03:44.673581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:44.672927 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:03:44.747516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:44.736783 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:03:44.747516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:44.736840 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 22:03:45.251996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:45.251899 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" event={"ID":"3b7d5514-83c1-488f-813a-542407800d2a","Type":"ContainerStarted","Data":"938e6b90ab063d522f1b864a7d24d1d6706adba1ebf68e39d64d652401f4b261"} Oct 30 22:03:45.251996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:45.251968 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" event={"ID":"3b7d5514-83c1-488f-813a-542407800d2a","Type":"ContainerStarted","Data":"ba62fce347d99945751bafca1a9c2ffe19eb3c078b23c589d0bec5d95862ffdd"} Oct 30 22:03:45.380786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:45.380723 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" podStartSLOduration=2.380704721 podStartE2EDuration="2.380704721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:45.373965512 +0000 UTC m=+1274.055115406" watchObservedRunningTime="2026-10-30 22:03:45.380704721 +0000 UTC m=+1274.061854615" Oct 30 22:03:45.675249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:45.675211 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:03:45.675861 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:45.675827 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:03:45.715358 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:03:45.715326 4933 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7d5514_83c1_488f_813a_542407800d2a.slice/crio-938e6b90ab063d522f1b864a7d24d1d6706adba1ebf68e39d64d652401f4b261.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:03:45.738328 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:45.738281 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:03:45.738420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:45.738327 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 22:03:46.288477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:46.288441 4933 generic.go:356] "Generic (PLEG): container finished" podID="3b7d5514-83c1-488f-813a-542407800d2a" containerID="938e6b90ab063d522f1b864a7d24d1d6706adba1ebf68e39d64d652401f4b261" exitCode=0 Oct 30 22:03:46.288836 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:46.288813 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" event={"ID":"3b7d5514-83c1-488f-813a-542407800d2a","Type":"ContainerDied","Data":"938e6b90ab063d522f1b864a7d24d1d6706adba1ebf68e39d64d652401f4b261"} Oct 30 22:03:46.683894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:46.683802 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:03:46.684442 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:46.684410 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:03:46.741515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:46.737997 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:03:46.741515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:46.738056 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 22:03:47.614674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.614639 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:47.671904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.671869 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 22:03:47.672089 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.671931 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 22:03:47.711155 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.711115 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-xks-ca\") pod \"3b7d5514-83c1-488f-813a-542407800d2a\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " Oct 30 22:03:47.711656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.711270 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7d5514-83c1-488f-813a-542407800d2a-kube-api-access-pqv59\" (UniqueName: \"kubernetes.io/projected/3b7d5514-83c1-488f-813a-542407800d2a-kube-api-access-pqv59\") pod \"3b7d5514-83c1-488f-813a-542407800d2a\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " Oct 30 22:03:47.711656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.711301 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-openshift-ca\") pod \"3b7d5514-83c1-488f-813a-542407800d2a\" (UID: \"3b7d5514-83c1-488f-813a-542407800d2a\") " Oct 30 22:03:47.711776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.711761 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-openshift-ca" pod "3b7d5514-83c1-488f-813a-542407800d2a" (UID: "3b7d5514-83c1-488f-813a-542407800d2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:47.711947 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.711929 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:47.713814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.713786 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-xks-ca" pod "3b7d5514-83c1-488f-813a-542407800d2a" (UID: "3b7d5514-83c1-488f-813a-542407800d2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:47.714144 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.714118 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7d5514-83c1-488f-813a-542407800d2a-kube-api-access-pqv59" pod "3b7d5514-83c1-488f-813a-542407800d2a" (UID: "3b7d5514-83c1-488f-813a-542407800d2a"). InnerVolumeSpecName "kube-api-access-pqv59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:47.759510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.737170 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:03:47.759510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.737237 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 22:03:47.813665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.813631 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqv59\" (UniqueName: \"kubernetes.io/projected/3b7d5514-83c1-488f-813a-542407800d2a-kube-api-access-pqv59\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:47.813665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:47.813673 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b7d5514-83c1-488f-813a-542407800d2a-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:48.316275 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:48.316239 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" event={"ID":"3b7d5514-83c1-488f-813a-542407800d2a","Type":"ContainerDied","Data":"ba62fce347d99945751bafca1a9c2ffe19eb3c078b23c589d0bec5d95862ffdd"} Oct 30 22:03:48.316275 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:48.316259 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn" Oct 30 22:03:48.316570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:48.316295 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba62fce347d99945751bafca1a9c2ffe19eb3c078b23c589d0bec5d95862ffdd" Oct 30 22:03:49.092894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:49.092818 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:49.094072 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:49.093630 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:49.097673 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:49.097645 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:03:49.115564 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:49.115484 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:49.116139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:49.116118 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:49.117550 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:49.117524 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:03:51.889233 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.889202 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm"] Oct 30 22:03:51.890126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.890097 4933 cpu_manager.go:414] "containerMap: removing container" podUID="3b7d5514-83c1-488f-813a-542407800d2a" containerName="revoke-keys" Oct 30 22:03:51.890126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.890126 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7d5514-83c1-488f-813a-542407800d2a" containerName="revoke-keys" Oct 30 22:03:51.890330 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.890315 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7d5514-83c1-488f-813a-542407800d2a" containerName="revoke-keys" Oct 30 22:03:51.894945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.894926 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:51.948095 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.948065 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:51.948287 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.948181 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:51.948352 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.948305 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:51.951370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.951348 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm"] Oct 30 22:03:51.996677 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:51.996651 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:03:52.058527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.058478 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57tzk\" (UniqueName: \"kubernetes.io/projected/cdb2dadf-3719-45ea-b6de-6e031de9e683-kube-api-access-57tzk\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.058651 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.058630 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.058704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.058666 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.159624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.159541 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.159624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.159599 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.159837 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.159821 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57tzk\" (UniqueName: \"kubernetes.io/projected/cdb2dadf-3719-45ea-b6de-6e031de9e683-kube-api-access-57tzk\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.160318 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.160301 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.162061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.162044 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.218187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.218162 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57tzk\" (UniqueName: \"kubernetes.io/projected/cdb2dadf-3719-45ea-b6de-6e031de9e683-kube-api-access-57tzk\") pod \"maas-api-revoke-sub-d9896b16-zp5dm\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.510318 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.510239 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:52.898623 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:03:52.898588 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb2dadf_3719_45ea_b6de_6e031de9e683.slice/crio-21c314f589339e0044dc04e9b6887c3b2af4fc607c926d4299088292449678e8 WatchSource:0}: Error finding container 21c314f589339e0044dc04e9b6887c3b2af4fc607c926d4299088292449678e8: Status 404 returned error can't find the container with id 21c314f589339e0044dc04e9b6887c3b2af4fc607c926d4299088292449678e8 Oct 30 22:03:52.899645 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:52.899602 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm"] Oct 30 22:03:53.054696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.054665 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q"] Oct 30 22:03:53.059080 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.059053 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8"] Oct 30 22:03:53.104174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.104152 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1386df0b-7vp4q"] Oct 30 22:03:53.107979 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.107961 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e630f73-dvkp8"] Oct 30 22:03:53.468689 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.468661 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0261f241-2920-497f-92cd-a728f6ca3a12" path="/var/lib/kubelet/pods/0261f241-2920-497f-92cd-a728f6ca3a12/volumes" Oct 30 22:03:53.469099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.469078 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a5b995-ad9d-4102-92eb-0e0b5d094cd3" path="/var/lib/kubelet/pods/60a5b995-ad9d-4102-92eb-0e0b5d094cd3/volumes" Oct 30 22:03:53.479481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.479454 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" event={"ID":"cdb2dadf-3719-45ea-b6de-6e031de9e683","Type":"ContainerStarted","Data":"48d94768aa2d3851d4de52a465f60febe9185a89f4f0eeaf4aa71932c2b30f74"} Oct 30 22:03:53.479615 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.479521 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" event={"ID":"cdb2dadf-3719-45ea-b6de-6e031de9e683","Type":"ContainerStarted","Data":"21c314f589339e0044dc04e9b6887c3b2af4fc607c926d4299088292449678e8"} Oct 30 22:03:53.608474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:53.608425 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" podStartSLOduration=2.608408078 podStartE2EDuration="2.608408078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:53.605076261 +0000 UTC m=+1282.286226154" watchObservedRunningTime="2026-10-30 22:03:53.608408078 +0000 UTC m=+1282.289557973" Oct 30 22:03:54.498858 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:54.498819 4933 generic.go:356] "Generic (PLEG): container finished" podID="cdb2dadf-3719-45ea-b6de-6e031de9e683" containerID="48d94768aa2d3851d4de52a465f60febe9185a89f4f0eeaf4aa71932c2b30f74" exitCode=0 Oct 30 22:03:54.499281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:54.498904 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" event={"ID":"cdb2dadf-3719-45ea-b6de-6e031de9e683","Type":"ContainerDied","Data":"48d94768aa2d3851d4de52a465f60febe9185a89f4f0eeaf4aa71932c2b30f74"} Oct 30 22:03:55.675382 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.675355 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:55.705378 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.705349 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-xks-ca\") pod \"cdb2dadf-3719-45ea-b6de-6e031de9e683\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " Oct 30 22:03:55.705543 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.705428 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-openshift-ca\") pod \"cdb2dadf-3719-45ea-b6de-6e031de9e683\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " Oct 30 22:03:55.705629 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.705612 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb2dadf-3719-45ea-b6de-6e031de9e683-kube-api-access-57tzk\" (UniqueName: \"kubernetes.io/projected/cdb2dadf-3719-45ea-b6de-6e031de9e683-kube-api-access-57tzk\") pod \"cdb2dadf-3719-45ea-b6de-6e031de9e683\" (UID: \"cdb2dadf-3719-45ea-b6de-6e031de9e683\") " Oct 30 22:03:55.705841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.705819 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-openshift-ca" pod "cdb2dadf-3719-45ea-b6de-6e031de9e683" (UID: "cdb2dadf-3719-45ea-b6de-6e031de9e683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:03:55.706140 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.706120 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:55.707817 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.707791 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb2dadf-3719-45ea-b6de-6e031de9e683-kube-api-access-57tzk" pod "cdb2dadf-3719-45ea-b6de-6e031de9e683" (UID: "cdb2dadf-3719-45ea-b6de-6e031de9e683"). InnerVolumeSpecName "kube-api-access-57tzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:03:55.707946 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.707930 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-xks-ca" pod "cdb2dadf-3719-45ea-b6de-6e031de9e683" (UID: "cdb2dadf-3719-45ea-b6de-6e031de9e683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:03:55.807208 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.807180 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57tzk\" (UniqueName: \"kubernetes.io/projected/cdb2dadf-3719-45ea-b6de-6e031de9e683-kube-api-access-57tzk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:55.807208 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:55.807207 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdb2dadf-3719-45ea-b6de-6e031de9e683-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:03:56.516189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:56.516155 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" event={"ID":"cdb2dadf-3719-45ea-b6de-6e031de9e683","Type":"ContainerDied","Data":"21c314f589339e0044dc04e9b6887c3b2af4fc607c926d4299088292449678e8"} Oct 30 22:03:56.516189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:56.516179 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm" Oct 30 22:03:56.516399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:56.516206 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c314f589339e0044dc04e9b6887c3b2af4fc607c926d4299088292449678e8" Oct 30 22:03:58.546540 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:03:58.546476 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:03:59.879444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.879410 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5"] Oct 30 22:03:59.880036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.880020 4933 cpu_manager.go:414] "containerMap: removing container" podUID="cdb2dadf-3719-45ea-b6de-6e031de9e683" containerName="revoke-keys" Oct 30 22:03:59.880098 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.880038 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb2dadf-3719-45ea-b6de-6e031de9e683" containerName="revoke-keys" Oct 30 22:03:59.880160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.880150 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb2dadf-3719-45ea-b6de-6e031de9e683" containerName="revoke-keys" Oct 30 22:03:59.883791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.883773 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:03:59.942701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.942671 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5"] Oct 30 22:03:59.959247 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.959217 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjqft\" (UniqueName: \"kubernetes.io/projected/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-kube-api-access-zjqft\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:03:59.959406 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.959371 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:03:59.959482 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.959473 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:03:59.986666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.986545 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:03:59.986666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.986569 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:03:59.986666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.986626 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:03:59.988715 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:03:59.988697 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:04:00.061036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.061014 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:00.061175 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.061133 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjqft\" (UniqueName: \"kubernetes.io/projected/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-kube-api-access-zjqft\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:00.061276 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.061262 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:00.061981 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.061914 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:00.063672 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.063649 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:00.119277 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.119251 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjqft\" (UniqueName: \"kubernetes.io/projected/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-kube-api-access-zjqft\") pod \"maas-api-revoke-sub-0e4d0522-nqzv5\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:00.201475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.201410 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:00.583568 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:00.583542 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5"] Oct 30 22:04:00.587423 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:00.587395 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc8d3e4_62b8_4d4b_9bda_c23b635f36a2.slice/crio-b7c52cd3cee5faf121c1b042963ec83ae935d384c89745bd3fc52429357c6348 WatchSource:0}: Error finding container b7c52cd3cee5faf121c1b042963ec83ae935d384c89745bd3fc52429357c6348: Status 404 returned error can't find the container with id b7c52cd3cee5faf121c1b042963ec83ae935d384c89745bd3fc52429357c6348 Oct 30 22:04:01.562303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:01.562272 4933 generic.go:356] "Generic (PLEG): container finished" podID="5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" containerID="b37c09a19e5c0a81566ff9b11ab6ea5b6be1777481ff43cbf9b04b6f5677edc3" exitCode=0 Oct 30 22:04:01.562779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:01.562361 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" event={"ID":"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2","Type":"ContainerDied","Data":"b37c09a19e5c0a81566ff9b11ab6ea5b6be1777481ff43cbf9b04b6f5677edc3"} Oct 30 22:04:01.562779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:01.562403 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" event={"ID":"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2","Type":"ContainerStarted","Data":"b7c52cd3cee5faf121c1b042963ec83ae935d384c89745bd3fc52429357c6348"} Oct 30 22:04:02.050144 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.050108 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl"] Oct 30 22:04:02.055567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.055541 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d917d11-xx2gl"] Oct 30 22:04:02.700378 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.700337 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8"] Oct 30 22:04:02.704652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.704631 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.755210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.755183 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:02.763980 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.763955 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8"] Oct 30 22:04:02.795179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.795150 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.795334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.795186 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.795402 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.795354 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5ns\" (UniqueName: \"kubernetes.io/projected/be7da15a-872a-4552-85dd-dd465812ec50-kube-api-access-sw5ns\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.896413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.896333 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-openshift-ca\") pod \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " Oct 30 22:04:02.896576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.896460 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-kube-api-access-zjqft\" (UniqueName: \"kubernetes.io/projected/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-kube-api-access-zjqft\") pod \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " Oct 30 22:04:02.896576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.896561 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-xks-ca\") pod \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\" (UID: \"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2\") " Oct 30 22:04:02.896796 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.896768 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-openshift-ca" pod "5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" (UID: "5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:02.897035 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.897012 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.897120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.897054 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.897336 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.897137 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5ns\" (UniqueName: \"kubernetes.io/projected/be7da15a-872a-4552-85dd-dd465812ec50-kube-api-access-sw5ns\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.897336 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.897233 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.897849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.897821 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.898901 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.898870 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-kube-api-access-zjqft" pod "5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" (UID: "5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2"). InnerVolumeSpecName "kube-api-access-zjqft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:02.899041 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.899021 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-xks-ca" pod "5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" (UID: "5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:02.899674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.899653 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.957223 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.957194 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5ns\" (UniqueName: \"kubernetes.io/projected/be7da15a-872a-4552-85dd-dd465812ec50-kube-api-access-sw5ns\") pod \"maas-api-revoke-sub-35ebeaac-gxwp8\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:02.998269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.998238 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjqft\" (UniqueName: \"kubernetes.io/projected/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-kube-api-access-zjqft\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:02.998269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:02.998268 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:03.020975 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:03.020947 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:03.413662 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:03.413633 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7da15a_872a_4552_85dd_dd465812ec50.slice/crio-d992eb0c7796cc8c9143313c0503e027645070c9c2383df78f4c022538edbdcf WatchSource:0}: Error finding container d992eb0c7796cc8c9143313c0503e027645070c9c2383df78f4c022538edbdcf: Status 404 returned error can't find the container with id d992eb0c7796cc8c9143313c0503e027645070c9c2383df78f4c022538edbdcf Oct 30 22:04:03.415016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:03.414993 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8"] Oct 30 22:04:03.466868 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:03.466843 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31c1bf5-c985-4791-99b0-d617606f2cc7" path="/var/lib/kubelet/pods/e31c1bf5-c985-4791-99b0-d617606f2cc7/volumes" Oct 30 22:04:03.584595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:03.584574 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" Oct 30 22:04:03.584738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:03.584609 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5" event={"ID":"5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2","Type":"ContainerDied","Data":"b7c52cd3cee5faf121c1b042963ec83ae935d384c89745bd3fc52429357c6348"} Oct 30 22:04:03.584738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:03.584664 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c52cd3cee5faf121c1b042963ec83ae935d384c89745bd3fc52429357c6348" Oct 30 22:04:03.586262 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:03.586228 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" event={"ID":"be7da15a-872a-4552-85dd-dd465812ec50","Type":"ContainerStarted","Data":"d992eb0c7796cc8c9143313c0503e027645070c9c2383df78f4c022538edbdcf"} Oct 30 22:04:04.595510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:04.595403 4933 generic.go:356] "Generic (PLEG): container finished" podID="be7da15a-872a-4552-85dd-dd465812ec50" containerID="db090e405d4144362fae730c97f702a81f383302ba4cc7d70c535ccca43c3863" exitCode=0 Oct 30 22:04:04.595873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:04.595501 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" event={"ID":"be7da15a-872a-4552-85dd-dd465812ec50","Type":"ContainerDied","Data":"db090e405d4144362fae730c97f702a81f383302ba4cc7d70c535ccca43c3863"} Oct 30 22:04:05.764364 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.764323 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:05.838016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.837989 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-openshift-ca\") pod \"be7da15a-872a-4552-85dd-dd465812ec50\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " Oct 30 22:04:05.838166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.838110 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-xks-ca\") pod \"be7da15a-872a-4552-85dd-dd465812ec50\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " Oct 30 22:04:05.838166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.838156 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7da15a-872a-4552-85dd-dd465812ec50-kube-api-access-sw5ns\" (UniqueName: \"kubernetes.io/projected/be7da15a-872a-4552-85dd-dd465812ec50-kube-api-access-sw5ns\") pod \"be7da15a-872a-4552-85dd-dd465812ec50\" (UID: \"be7da15a-872a-4552-85dd-dd465812ec50\") " Oct 30 22:04:05.838328 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.838306 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-openshift-ca" pod "be7da15a-872a-4552-85dd-dd465812ec50" (UID: "be7da15a-872a-4552-85dd-dd465812ec50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:05.838692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.838671 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:05.841052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.841025 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-xks-ca" pod "be7da15a-872a-4552-85dd-dd465812ec50" (UID: "be7da15a-872a-4552-85dd-dd465812ec50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:05.841052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.841044 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7da15a-872a-4552-85dd-dd465812ec50-kube-api-access-sw5ns" pod "be7da15a-872a-4552-85dd-dd465812ec50" (UID: "be7da15a-872a-4552-85dd-dd465812ec50"). InnerVolumeSpecName "kube-api-access-sw5ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:05.940051 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.939972 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be7da15a-872a-4552-85dd-dd465812ec50-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:05.940245 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:05.940227 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5ns\" (UniqueName: \"kubernetes.io/projected/be7da15a-872a-4552-85dd-dd465812ec50-kube-api-access-sw5ns\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:06.612018 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:06.611994 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" Oct 30 22:04:06.612191 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:06.612024 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8" event={"ID":"be7da15a-872a-4552-85dd-dd465812ec50","Type":"ContainerDied","Data":"d992eb0c7796cc8c9143313c0503e027645070c9c2383df78f4c022538edbdcf"} Oct 30 22:04:06.612191 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:06.612075 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d992eb0c7796cc8c9143313c0503e027645070c9c2383df78f4c022538edbdcf" Oct 30 22:04:07.530476 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.530446 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw"] Oct 30 22:04:07.531029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.531016 4933 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" containerName="revoke-keys" Oct 30 22:04:07.531087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.531029 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" containerName="revoke-keys" Oct 30 22:04:07.531087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.531050 4933 cpu_manager.go:414] "containerMap: removing container" podUID="be7da15a-872a-4552-85dd-dd465812ec50" containerName="revoke-keys" Oct 30 22:04:07.531087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.531056 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7da15a-872a-4552-85dd-dd465812ec50" containerName="revoke-keys" Oct 30 22:04:07.531197 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.531187 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7da15a-872a-4552-85dd-dd465812ec50" containerName="revoke-keys" Oct 30 22:04:07.531240 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.531204 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" containerName="revoke-keys" Oct 30 22:04:07.534906 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.534888 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.592654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.592622 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw"] Oct 30 22:04:07.637183 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.637153 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:07.637349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.637242 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:07.637349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.637252 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:07.641505 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.641464 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:04:07.665460 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.665427 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.665628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.665571 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2k86\" (UniqueName: \"kubernetes.io/projected/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-kube-api-access-t2k86\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.665763 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.665741 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.767590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.767561 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.767780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.767661 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2k86\" (UniqueName: \"kubernetes.io/projected/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-kube-api-access-t2k86\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.767842 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.767779 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.768642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.768617 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.770020 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.770001 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.826028 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.825971 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2k86\" (UniqueName: \"kubernetes.io/projected/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-kube-api-access-t2k86\") pod \"maas-api-revoke-sub-126845c2-rdprw\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:07.849249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:07.849215 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:08.052815 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.052783 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt"] Oct 30 22:04:08.059694 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.059673 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc"] Oct 30 22:04:08.065809 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.065794 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46489f2d-f4gdt"] Oct 30 22:04:08.071762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.071739 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9202767f-lllfc"] Oct 30 22:04:08.234297 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:08.234268 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1f86c0_be28_4668_8a9e_0e8d71a94c77.slice/crio-24440587d21e64a3d9c80f5cd29876e8824f5ca3dcb6f3826ac70edbadd834c6 WatchSource:0}: Error finding container 24440587d21e64a3d9c80f5cd29876e8824f5ca3dcb6f3826ac70edbadd834c6: Status 404 returned error can't find the container with id 24440587d21e64a3d9c80f5cd29876e8824f5ca3dcb6f3826ac70edbadd834c6 Oct 30 22:04:08.236856 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.236829 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw"] Oct 30 22:04:08.629917 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.629881 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" event={"ID":"cc1f86c0-be28-4668-8a9e-0e8d71a94c77","Type":"ContainerStarted","Data":"c1e4af932a6efb4cf96a494b62114245ad95257ed16c64e24795c9d96dc74d69"} Oct 30 22:04:08.630317 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.629953 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" event={"ID":"cc1f86c0-be28-4668-8a9e-0e8d71a94c77","Type":"ContainerStarted","Data":"24440587d21e64a3d9c80f5cd29876e8824f5ca3dcb6f3826ac70edbadd834c6"} Oct 30 22:04:08.748781 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:08.748734 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" podStartSLOduration=1.748720312 podStartE2EDuration="1.748720312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:08.74628557 +0000 UTC m=+1297.427435466" watchObservedRunningTime="2026-10-30 22:04:08.748720312 +0000 UTC m=+1297.429870204" Oct 30 22:04:09.053143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.053115 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6"] Oct 30 22:04:09.059178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.059155 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8adb9c8-zqtc6"] Oct 30 22:04:09.312302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.312234 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv"] Oct 30 22:04:09.316452 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.316428 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.425303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.425272 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv"] Oct 30 22:04:09.468393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.468348 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c18b75-ce39-4fc0-a760-051d66ae16ae" path="/var/lib/kubelet/pods/19c18b75-ce39-4fc0-a760-051d66ae16ae/volumes" Oct 30 22:04:09.468955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.468933 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48c73ad-e333-4fc6-b215-1c087df592d6" path="/var/lib/kubelet/pods/d48c73ad-e333-4fc6-b215-1c087df592d6/volumes" Oct 30 22:04:09.469405 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.469387 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9189e58-8445-4bc0-936f-a2692d037906" path="/var/lib/kubelet/pods/e9189e58-8445-4bc0-936f-a2692d037906/volumes" Oct 30 22:04:09.494395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494364 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06842ac-3593-45e2-b9e1-e40f53738576-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494553 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494604 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06842ac-3593-45e2-b9e1-e40f53738576-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494679 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494754 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494739 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494793 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494871 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494910 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgngk\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-kube-api-access-wgngk\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.494982 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.494931 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596340 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596419 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596454 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgngk\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-kube-api-access-wgngk\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596560 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596631 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06842ac-3593-45e2-b9e1-e40f53738576-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596782 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596799 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596871 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06842ac-3593-45e2-b9e1-e40f53738576-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.596922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596905 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.597020 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.596965 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.597067 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.597031 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.597340 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.597262 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.597470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.597401 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.597642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.597624 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06842ac-3593-45e2-b9e1-e40f53738576-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.599000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.598977 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.599265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.599246 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06842ac-3593-45e2-b9e1-e40f53738576-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.645765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.645727 4933 generic.go:356] "Generic (PLEG): container finished" podID="cc1f86c0-be28-4668-8a9e-0e8d71a94c77" containerID="c1e4af932a6efb4cf96a494b62114245ad95257ed16c64e24795c9d96dc74d69" exitCode=0 Oct 30 22:04:09.646189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.645814 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" event={"ID":"cc1f86c0-be28-4668-8a9e-0e8d71a94c77","Type":"ContainerDied","Data":"c1e4af932a6efb4cf96a494b62114245ad95257ed16c64e24795c9d96dc74d69"} Oct 30 22:04:09.654420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.654396 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.654844 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.654826 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgngk\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-kube-api-access-wgngk\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-987mv\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:09.820447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.820416 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:04:09.935950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:09.935876 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:10.378108 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:10.378075 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06842ac_3593_45e2_b9e1_e40f53738576.slice/crio-cd40eb49253ca764a49980d40a8661ee1144f90f541c75ee5bb39052bca3d8cc WatchSource:0}: Error finding container cd40eb49253ca764a49980d40a8661ee1144f90f541c75ee5bb39052bca3d8cc: Status 404 returned error can't find the container with id cd40eb49253ca764a49980d40a8661ee1144f90f541c75ee5bb39052bca3d8cc Oct 30 22:04:10.378411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.378389 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv"] Oct 30 22:04:10.380940 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.380911 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:10.381018 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.380972 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:10.381018 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.381004 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:10.660042 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.659940 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" event={"ID":"f06842ac-3593-45e2-b9e1-e40f53738576","Type":"ContainerStarted","Data":"6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026"} Oct 30 22:04:10.660042 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.660023 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" event={"ID":"f06842ac-3593-45e2-b9e1-e40f53738576","Type":"ContainerStarted","Data":"cd40eb49253ca764a49980d40a8661ee1144f90f541c75ee5bb39052bca3d8cc"} Oct 30 22:04:10.786193 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.784910 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podStartSLOduration=1.7848883070000001 podStartE2EDuration="1.784888307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:10.780986132 +0000 UTC m=+1299.462136059" watchObservedRunningTime="2026-10-30 22:04:10.784888307 +0000 UTC m=+1299.466038201" Oct 30 22:04:10.922688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.922608 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:10.938523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:10.936515 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:11.038633 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.036274 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-openshift-ca\") pod \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " Oct 30 22:04:11.038633 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.036390 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-xks-ca\") pod \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " Oct 30 22:04:11.038633 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.036473 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-kube-api-access-t2k86\" (UniqueName: \"kubernetes.io/projected/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-kube-api-access-t2k86\") pod \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\" (UID: \"cc1f86c0-be28-4668-8a9e-0e8d71a94c77\") " Oct 30 22:04:11.038633 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.037983 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-openshift-ca" pod "cc1f86c0-be28-4668-8a9e-0e8d71a94c77" (UID: "cc1f86c0-be28-4668-8a9e-0e8d71a94c77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:11.174445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.170655 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:11.245168 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.245096 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-kube-api-access-t2k86" pod "cc1f86c0-be28-4668-8a9e-0e8d71a94c77" (UID: "cc1f86c0-be28-4668-8a9e-0e8d71a94c77"). InnerVolumeSpecName "kube-api-access-t2k86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:11.275371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.272726 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2k86\" (UniqueName: \"kubernetes.io/projected/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-kube-api-access-t2k86\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:11.275371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.275117 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-xks-ca" pod "cc1f86c0-be28-4668-8a9e-0e8d71a94c77" (UID: "cc1f86c0-be28-4668-8a9e-0e8d71a94c77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:11.386521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.382657 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f86c0-be28-4668-8a9e-0e8d71a94c77-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:11.964667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.964618 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:11.965190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:11.964708 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:12.093513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.078552 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" event={"ID":"cc1f86c0-be28-4668-8a9e-0e8d71a94c77","Type":"ContainerDied","Data":"24440587d21e64a3d9c80f5cd29876e8824f5ca3dcb6f3826ac70edbadd834c6"} Oct 30 22:04:12.093513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.078658 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24440587d21e64a3d9c80f5cd29876e8824f5ca3dcb6f3826ac70edbadd834c6" Oct 30 22:04:12.093513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.079381 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw" Oct 30 22:04:12.619522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.619072 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.623514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.620302 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.686517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.682249 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.699516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.696609 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.748515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.748153 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:12.908576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.907573 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:12.944899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.944861 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:12.945180 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.945150 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:12.945382 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:12.945359 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.007665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.007620 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.131513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.123564 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.175715 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.175573 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.205529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.203579 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.210520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.206646 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.429571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.426565 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.594530 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.551029 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.622554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.621331 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.622554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.621822 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:04:13.646532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.641600 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:13.646532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.642023 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:04:13.692572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.686617 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.692572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.687102 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:04:13.733911 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.718609 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.733911 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.719026 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:04:13.773749 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.773706 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.809587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.792606 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:04:13.908517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.907726 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.951589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.909524 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:04:13.951589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.947588 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:13.951589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.948750 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:13.951589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.948793 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:13.951589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:13.948892 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:04:14.058244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.055591 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.058244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.055945 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:04:14.213722 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.213617 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:14.217383 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.215130 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:04:14.262439 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.262398 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.400534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.328721 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:14.503775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.373985 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:04:14.504159 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.504141 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:04:14.504318 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.504307 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:04:14.504471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.504461 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:04:14.530121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.432110 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.531387 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.531355 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:04:14.798514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.621725 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:14.999087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.666805 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:14.999817 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.798855 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.000125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.798921 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:15.000422 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.801394 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.000806 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:14.986422 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:15.000967 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.000940 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:15.001154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.001132 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.065205 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.064780 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.156511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.130596 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.354964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.352414 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:04:15.425510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.414948 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.425510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.415751 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:04:15.425510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.416762 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:04:15.445974 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.418167 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.470533 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.449015 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.507664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.506860 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:15.613627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.607400 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.721837 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.721803 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:04:15.722030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.721966 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:04:15.722030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.722025 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:04:15.722143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.722073 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:04:15.722143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.722120 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:04:15.722245 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.722166 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:04:15.722245 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.722211 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:04:15.724446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.724414 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:04:15.961954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.961870 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:15.962204 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:15.962174 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:16.977211 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:16.976870 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:16.977211 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:16.976935 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:17.783310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.783267 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz"] Oct 30 22:04:17.783988 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.783969 4933 cpu_manager.go:414] "containerMap: removing container" podUID="cc1f86c0-be28-4668-8a9e-0e8d71a94c77" containerName="revoke-keys" Oct 30 22:04:17.783988 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.783989 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1f86c0-be28-4668-8a9e-0e8d71a94c77" containerName="revoke-keys" Oct 30 22:04:17.784574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.784191 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1f86c0-be28-4668-8a9e-0e8d71a94c77" containerName="revoke-keys" Oct 30 22:04:17.787776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.787747 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:17.841421 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.841223 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:17.841421 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.841321 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:17.841891 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.841859 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:17.842840 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.842801 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:04:17.870087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.870040 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz"] Oct 30 22:04:17.909792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.909753 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:17.909938 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.909816 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx9jb\" (UniqueName: \"kubernetes.io/projected/88eb1189-4cd8-4546-affb-042fa0f0073f-kube-api-access-xx9jb\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:17.909994 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.909972 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:17.952514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.936821 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:17.952514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:17.936881 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:18.011557 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.011527 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:18.011991 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.011572 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx9jb\" (UniqueName: \"kubernetes.io/projected/88eb1189-4cd8-4546-affb-042fa0f0073f-kube-api-access-xx9jb\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:18.011991 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.011625 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:18.012278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.012256 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:18.014292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.014267 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:18.068418 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.068302 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r"] Oct 30 22:04:18.075905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.075847 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35111e14-xv22r"] Oct 30 22:04:18.076924 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.076900 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx9jb\" (UniqueName: \"kubernetes.io/projected/88eb1189-4cd8-4546-affb-042fa0f0073f-kube-api-access-xx9jb\") pod \"maas-api-revoke-sub-cb459a08-tgllz\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:18.080177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.080156 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7"] Oct 30 22:04:18.084751 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.084729 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.101257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.101241 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:18.165671 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.156623 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7"] Oct 30 22:04:18.215932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.213580 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.215932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.213636 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.215932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.213767 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48v8z\" (UniqueName: \"kubernetes.io/projected/9115f3ae-c5af-4754-8e06-bf1be900e777-kube-api-access-48v8z\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.316519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.314648 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48v8z\" (UniqueName: \"kubernetes.io/projected/9115f3ae-c5af-4754-8e06-bf1be900e777-kube-api-access-48v8z\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.316519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.314734 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.316519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.314773 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.316818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.316610 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.318848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.318769 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.377535 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.376999 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48v8z\" (UniqueName: \"kubernetes.io/projected/9115f3ae-c5af-4754-8e06-bf1be900e777-kube-api-access-48v8z\") pod \"maas-api-revoke-sub-61b16424-smwn7\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.401586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.401557 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:18.583240 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:18.583195 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eb1189_4cd8_4546_affb_042fa0f0073f.slice/crio-1e070bc817548d0d421b19b637e2a0b46abbddb9dde611fbeccd5cb7f983ff82 WatchSource:0}: Error finding container 1e070bc817548d0d421b19b637e2a0b46abbddb9dde611fbeccd5cb7f983ff82: Status 404 returned error can't find the container with id 1e070bc817548d0d421b19b637e2a0b46abbddb9dde611fbeccd5cb7f983ff82 Oct 30 22:04:18.588446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.588086 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz"] Oct 30 22:04:18.879909 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:18.879879 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9115f3ae_c5af_4754_8e06_bf1be900e777.slice/crio-7b7795e53ec96911f045f0042b029bef17a4563438bc0567128c974c54e47f3e WatchSource:0}: Error finding container 7b7795e53ec96911f045f0042b029bef17a4563438bc0567128c974c54e47f3e: Status 404 returned error can't find the container with id 7b7795e53ec96911f045f0042b029bef17a4563438bc0567128c974c54e47f3e Oct 30 22:04:18.882388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.882353 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7"] Oct 30 22:04:18.909457 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.909431 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" event={"ID":"88eb1189-4cd8-4546-affb-042fa0f0073f","Type":"ContainerStarted","Data":"1e070bc817548d0d421b19b637e2a0b46abbddb9dde611fbeccd5cb7f983ff82"} Oct 30 22:04:18.910772 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.910751 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" event={"ID":"9115f3ae-c5af-4754-8e06-bf1be900e777","Type":"ContainerStarted","Data":"7b7795e53ec96911f045f0042b029bef17a4563438bc0567128c974c54e47f3e"} Oct 30 22:04:18.954523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.953833 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:18.954523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:18.953885 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:19.072542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.072475 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl"] Oct 30 22:04:19.079798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.079772 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0de3460-vt4pl"] Oct 30 22:04:19.093668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.092707 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t"] Oct 30 22:04:19.098419 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.098395 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da03ebac-rnk5t"] Oct 30 22:04:19.473061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.472967 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257892d8-a41c-4b4b-96b8-b0985376d220" path="/var/lib/kubelet/pods/257892d8-a41c-4b4b-96b8-b0985376d220/volumes" Oct 30 22:04:19.473692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.473669 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9832f67b-f68e-4d85-873a-1f6db6005947" path="/var/lib/kubelet/pods/9832f67b-f68e-4d85-873a-1f6db6005947/volumes" Oct 30 22:04:19.474299 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.474277 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb802b5f-d726-4d17-ad7c-a26a76e9590f" path="/var/lib/kubelet/pods/eb802b5f-d726-4d17-ad7c-a26a76e9590f/volumes" Oct 30 22:04:19.922565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.922524 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" event={"ID":"9115f3ae-c5af-4754-8e06-bf1be900e777","Type":"ContainerStarted","Data":"03cf126c80184633ccf91e0caeb6438c67e64d7a0144ef4388ba4d901c0c961c"} Oct 30 22:04:19.924713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.924681 4933 generic.go:356] "Generic (PLEG): container finished" podID="88eb1189-4cd8-4546-affb-042fa0f0073f" containerID="b0e1eab495d8af99816012f5d8469a3be4094822f9ec51f2834fdc1401d31f93" exitCode=0 Oct 30 22:04:19.924854 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.924715 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" event={"ID":"88eb1189-4cd8-4546-affb-042fa0f0073f","Type":"ContainerDied","Data":"b0e1eab495d8af99816012f5d8469a3be4094822f9ec51f2834fdc1401d31f93"} Oct 30 22:04:19.936484 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.936450 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 22:04:19.936484 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:19.936507 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 22:04:20.043730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:20.043668 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" podStartSLOduration=2.043652764 podStartE2EDuration="2.043652764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:20.037828904 +0000 UTC m=+1308.718978799" watchObservedRunningTime="2026-10-30 22:04:20.043652764 +0000 UTC m=+1308.724802657" Oct 30 22:04:20.938428 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:20.938390 4933 generic.go:356] "Generic (PLEG): container finished" podID="9115f3ae-c5af-4754-8e06-bf1be900e777" containerID="03cf126c80184633ccf91e0caeb6438c67e64d7a0144ef4388ba4d901c0c961c" exitCode=0 Oct 30 22:04:20.939315 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:20.939283 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" event={"ID":"9115f3ae-c5af-4754-8e06-bf1be900e777","Type":"ContainerDied","Data":"03cf126c80184633ccf91e0caeb6438c67e64d7a0144ef4388ba4d901c0c961c"} Oct 30 22:04:20.946752 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:20.946110 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:20.947097 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:20.947076 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:20.948274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:20.948254 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:04:21.175311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.175287 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:21.298903 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.298870 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88eb1189-4cd8-4546-affb-042fa0f0073f-kube-api-access-xx9jb\" (UniqueName: \"kubernetes.io/projected/88eb1189-4cd8-4546-affb-042fa0f0073f-kube-api-access-xx9jb\") pod \"88eb1189-4cd8-4546-affb-042fa0f0073f\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " Oct 30 22:04:21.299106 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.298960 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-openshift-ca\") pod \"88eb1189-4cd8-4546-affb-042fa0f0073f\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " Oct 30 22:04:21.299183 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.299162 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-xks-ca\") pod \"88eb1189-4cd8-4546-affb-042fa0f0073f\" (UID: \"88eb1189-4cd8-4546-affb-042fa0f0073f\") " Oct 30 22:04:21.299555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.299527 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-openshift-ca" pod "88eb1189-4cd8-4546-affb-042fa0f0073f" (UID: "88eb1189-4cd8-4546-affb-042fa0f0073f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:21.300174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.300004 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.304082 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.304053 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eb1189-4cd8-4546-affb-042fa0f0073f-kube-api-access-xx9jb" pod "88eb1189-4cd8-4546-affb-042fa0f0073f" (UID: "88eb1189-4cd8-4546-affb-042fa0f0073f"). InnerVolumeSpecName "kube-api-access-xx9jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:21.304249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.304178 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-xks-ca" pod "88eb1189-4cd8-4546-affb-042fa0f0073f" (UID: "88eb1189-4cd8-4546-affb-042fa0f0073f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:21.401909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.401885 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx9jb\" (UniqueName: \"kubernetes.io/projected/88eb1189-4cd8-4546-affb-042fa0f0073f-kube-api-access-xx9jb\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.401909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.401909 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88eb1189-4cd8-4546-affb-042fa0f0073f-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:21.951935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.951899 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" event={"ID":"88eb1189-4cd8-4546-affb-042fa0f0073f","Type":"ContainerDied","Data":"1e070bc817548d0d421b19b637e2a0b46abbddb9dde611fbeccd5cb7f983ff82"} Oct 30 22:04:21.952437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.952002 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e070bc817548d0d421b19b637e2a0b46abbddb9dde611fbeccd5cb7f983ff82" Oct 30 22:04:21.952437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.951915 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz" Oct 30 22:04:21.983101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.983067 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql"] Oct 30 22:04:21.983945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.983918 4933 cpu_manager.go:414] "containerMap: removing container" podUID="88eb1189-4cd8-4546-affb-042fa0f0073f" containerName="revoke-keys" Oct 30 22:04:21.983945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.983942 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88eb1189-4cd8-4546-affb-042fa0f0073f" containerName="revoke-keys" Oct 30 22:04:21.984118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.984098 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="88eb1189-4cd8-4546-affb-042fa0f0073f" containerName="revoke-keys" Oct 30 22:04:21.988010 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:21.987988 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.059225 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.059198 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql"] Oct 30 22:04:22.110929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.110904 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkt6c\" (UniqueName: \"kubernetes.io/projected/cf9b4a49-9f70-4b43-b538-8ef7654eb444-kube-api-access-qkt6c\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.111102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.111084 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.111257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.111242 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.123439 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.123403 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:22.212251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.212183 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.212251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.212242 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt6c\" (UniqueName: \"kubernetes.io/projected/cf9b4a49-9f70-4b43-b538-8ef7654eb444-kube-api-access-qkt6c\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.212435 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.212337 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.212958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.212931 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.214747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.214729 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.270784 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.270758 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt6c\" (UniqueName: \"kubernetes.io/projected/cf9b4a49-9f70-4b43-b538-8ef7654eb444-kube-api-access-qkt6c\") pod \"maas-api-revoke-sub-3219452e-cw7ql\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.304466 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.304442 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:22.313130 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.313103 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9115f3ae-c5af-4754-8e06-bf1be900e777-kube-api-access-48v8z\" (UniqueName: \"kubernetes.io/projected/9115f3ae-c5af-4754-8e06-bf1be900e777-kube-api-access-48v8z\") pod \"9115f3ae-c5af-4754-8e06-bf1be900e777\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " Oct 30 22:04:22.313251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.313161 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-xks-ca\") pod \"9115f3ae-c5af-4754-8e06-bf1be900e777\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " Oct 30 22:04:22.313251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.313218 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-openshift-ca\") pod \"9115f3ae-c5af-4754-8e06-bf1be900e777\" (UID: \"9115f3ae-c5af-4754-8e06-bf1be900e777\") " Oct 30 22:04:22.313738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.313596 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-openshift-ca" pod "9115f3ae-c5af-4754-8e06-bf1be900e777" (UID: "9115f3ae-c5af-4754-8e06-bf1be900e777"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:22.314380 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.314355 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:22.315604 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.315576 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9115f3ae-c5af-4754-8e06-bf1be900e777-kube-api-access-48v8z" pod "9115f3ae-c5af-4754-8e06-bf1be900e777" (UID: "9115f3ae-c5af-4754-8e06-bf1be900e777"). InnerVolumeSpecName "kube-api-access-48v8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:22.315904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.315885 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-xks-ca" pod "9115f3ae-c5af-4754-8e06-bf1be900e777" (UID: "9115f3ae-c5af-4754-8e06-bf1be900e777"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:22.417164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.415974 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48v8z\" (UniqueName: \"kubernetes.io/projected/9115f3ae-c5af-4754-8e06-bf1be900e777-kube-api-access-48v8z\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:22.417164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.416008 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9115f3ae-c5af-4754-8e06-bf1be900e777-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:22.586913 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.586883 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n"] Oct 30 22:04:22.587537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.587521 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9115f3ae-c5af-4754-8e06-bf1be900e777" containerName="revoke-keys" Oct 30 22:04:22.588023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.587538 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9115f3ae-c5af-4754-8e06-bf1be900e777" containerName="revoke-keys" Oct 30 22:04:22.588023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.587669 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9115f3ae-c5af-4754-8e06-bf1be900e777" containerName="revoke-keys" Oct 30 22:04:22.592424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.592406 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.650308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.650280 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n"] Oct 30 22:04:22.699425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.699400 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql"] Oct 30 22:04:22.703289 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:22.703264 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9b4a49_9f70_4b43_b538_8ef7654eb444.slice/crio-ca60cd688f643ddefe0254cd17bec063b1c52d82c1b18c0e83a4d610601322ab WatchSource:0}: Error finding container ca60cd688f643ddefe0254cd17bec063b1c52d82c1b18c0e83a4d610601322ab: Status 404 returned error can't find the container with id ca60cd688f643ddefe0254cd17bec063b1c52d82c1b18c0e83a4d610601322ab Oct 30 22:04:22.720115 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.720088 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.720224 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.720127 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9mc\" (UniqueName: \"kubernetes.io/projected/fb08710c-0cab-423f-b093-4b22255fc6a7-kube-api-access-tg9mc\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.720324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.720302 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.822034 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.822005 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.822155 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.822040 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9mc\" (UniqueName: \"kubernetes.io/projected/fb08710c-0cab-423f-b093-4b22255fc6a7-kube-api-access-tg9mc\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.822214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.822163 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.822668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.822643 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.824653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.824635 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.879968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.879909 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9mc\" (UniqueName: \"kubernetes.io/projected/fb08710c-0cab-423f-b093-4b22255fc6a7-kube-api-access-tg9mc\") pod \"maas-api-revoke-sub-b9cb66d6-6pq4n\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.907366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.907341 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:22.969910 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.969733 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" event={"ID":"9115f3ae-c5af-4754-8e06-bf1be900e777","Type":"ContainerDied","Data":"7b7795e53ec96911f045f0042b029bef17a4563438bc0567128c974c54e47f3e"} Oct 30 22:04:22.969910 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.969785 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b7795e53ec96911f045f0042b029bef17a4563438bc0567128c974c54e47f3e" Oct 30 22:04:22.969910 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.969821 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7" Oct 30 22:04:22.972017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:22.971992 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" event={"ID":"cf9b4a49-9f70-4b43-b538-8ef7654eb444","Type":"ContainerStarted","Data":"ca60cd688f643ddefe0254cd17bec063b1c52d82c1b18c0e83a4d610601322ab"} Oct 30 22:04:23.291007 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:23.290981 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb08710c_0cab_423f_b093_4b22255fc6a7.slice/crio-e2f03c1afb1ad474fac57dd8687b4959e78c9779eded2d3d5ede0f5023c0d153 WatchSource:0}: Error finding container e2f03c1afb1ad474fac57dd8687b4959e78c9779eded2d3d5ede0f5023c0d153: Status 404 returned error can't find the container with id e2f03c1afb1ad474fac57dd8687b4959e78c9779eded2d3d5ede0f5023c0d153 Oct 30 22:04:23.292302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.292278 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n"] Oct 30 22:04:23.723767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.723736 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp"] Oct 30 22:04:23.728210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.728190 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.735106 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.735075 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.735214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.735143 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.735292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.735274 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx552\" (UniqueName: \"kubernetes.io/projected/3705c57b-5255-41c4-be8a-5add2186e611-kube-api-access-xx552\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.788280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.788251 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp"] Oct 30 22:04:23.836833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.836752 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx552\" (UniqueName: \"kubernetes.io/projected/3705c57b-5255-41c4-be8a-5add2186e611-kube-api-access-xx552\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.836985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.836952 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.837051 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.836999 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.837727 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.837663 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.839398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.839378 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.894315 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.894294 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx552\" (UniqueName: \"kubernetes.io/projected/3705c57b-5255-41c4-be8a-5add2186e611-kube-api-access-xx552\") pod \"maas-api-revoke-sub-b9fe2816-6vcrp\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:23.981309 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.981280 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" event={"ID":"fb08710c-0cab-423f-b093-4b22255fc6a7","Type":"ContainerStarted","Data":"15fb087cc3373bd378c1251ea3755f8190f23111546cd53cd7ac0e80c307bedc"} Oct 30 22:04:23.981686 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.981331 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" event={"ID":"fb08710c-0cab-423f-b093-4b22255fc6a7","Type":"ContainerStarted","Data":"e2f03c1afb1ad474fac57dd8687b4959e78c9779eded2d3d5ede0f5023c0d153"} Oct 30 22:04:23.983038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.983013 4933 generic.go:356] "Generic (PLEG): container finished" podID="cf9b4a49-9f70-4b43-b538-8ef7654eb444" containerID="1f085afa259f76b70ef613491ebfb82a901335cb5eb06cc46ebc273ada0126b1" exitCode=0 Oct 30 22:04:23.983149 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:23.983097 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" event={"ID":"cf9b4a49-9f70-4b43-b538-8ef7654eb444","Type":"ContainerDied","Data":"1f085afa259f76b70ef613491ebfb82a901335cb5eb06cc46ebc273ada0126b1"} Oct 30 22:04:24.045928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:24.045908 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:24.118279 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:24.118233 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" podStartSLOduration=2.118217499 podStartE2EDuration="2.118217499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:24.116094161 +0000 UTC m=+1312.797244057" watchObservedRunningTime="2026-10-30 22:04:24.118217499 +0000 UTC m=+1312.799367392" Oct 30 22:04:24.424512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:24.424470 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp"] Oct 30 22:04:24.425955 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:24.425931 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3705c57b_5255_41c4_be8a_5add2186e611.slice/crio-9a3fd4ff89d9b09c98fa69a3897ecbbbc76b31bb08dfae36e8c5aabe53d3ad7a WatchSource:0}: Error finding container 9a3fd4ff89d9b09c98fa69a3897ecbbbc76b31bb08dfae36e8c5aabe53d3ad7a: Status 404 returned error can't find the container with id 9a3fd4ff89d9b09c98fa69a3897ecbbbc76b31bb08dfae36e8c5aabe53d3ad7a Oct 30 22:04:24.992828 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:24.992801 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" event={"ID":"3705c57b-5255-41c4-be8a-5add2186e611","Type":"ContainerStarted","Data":"b690102398fefd437f472162939e582d26c95ddc3ebafebeb2cc62eb489cf55c"} Oct 30 22:04:24.993229 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:24.992860 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" event={"ID":"3705c57b-5255-41c4-be8a-5add2186e611","Type":"ContainerStarted","Data":"9a3fd4ff89d9b09c98fa69a3897ecbbbc76b31bb08dfae36e8c5aabe53d3ad7a"} Oct 30 22:04:24.994554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:24.994525 4933 generic.go:356] "Generic (PLEG): container finished" podID="fb08710c-0cab-423f-b093-4b22255fc6a7" containerID="15fb087cc3373bd378c1251ea3755f8190f23111546cd53cd7ac0e80c307bedc" exitCode=0 Oct 30 22:04:24.994648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:24.994607 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" event={"ID":"fb08710c-0cab-423f-b093-4b22255fc6a7","Type":"ContainerDied","Data":"15fb087cc3373bd378c1251ea3755f8190f23111546cd53cd7ac0e80c307bedc"} Oct 30 22:04:25.111343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.111252 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" podStartSLOduration=2.111237446 podStartE2EDuration="2.111237446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:25.109478281 +0000 UTC m=+1313.790628173" watchObservedRunningTime="2026-10-30 22:04:25.111237446 +0000 UTC m=+1313.792387338" Oct 30 22:04:25.177459 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.177437 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:25.255257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.255233 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-openshift-ca\") pod \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " Oct 30 22:04:25.255364 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.255294 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-xks-ca\") pod \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " Oct 30 22:04:25.255473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.255461 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9b4a49-9f70-4b43-b538-8ef7654eb444-kube-api-access-qkt6c\" (UniqueName: \"kubernetes.io/projected/cf9b4a49-9f70-4b43-b538-8ef7654eb444-kube-api-access-qkt6c\") pod \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\" (UID: \"cf9b4a49-9f70-4b43-b538-8ef7654eb444\") " Oct 30 22:04:25.255639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.255615 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-openshift-ca" pod "cf9b4a49-9f70-4b43-b538-8ef7654eb444" (UID: "cf9b4a49-9f70-4b43-b538-8ef7654eb444"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:25.255999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.255983 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.257630 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.257609 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-xks-ca" pod "cf9b4a49-9f70-4b43-b538-8ef7654eb444" (UID: "cf9b4a49-9f70-4b43-b538-8ef7654eb444"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:25.257743 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.257624 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9b4a49-9f70-4b43-b538-8ef7654eb444-kube-api-access-qkt6c" pod "cf9b4a49-9f70-4b43-b538-8ef7654eb444" (UID: "cf9b4a49-9f70-4b43-b538-8ef7654eb444"). InnerVolumeSpecName "kube-api-access-qkt6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:25.357004 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.356968 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkt6c\" (UniqueName: \"kubernetes.io/projected/cf9b4a49-9f70-4b43-b538-8ef7654eb444-kube-api-access-qkt6c\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:25.357004 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:25.356999 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9b4a49-9f70-4b43-b538-8ef7654eb444-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.006458 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.006434 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" Oct 30 22:04:26.006998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.006558 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql" event={"ID":"cf9b4a49-9f70-4b43-b538-8ef7654eb444","Type":"ContainerDied","Data":"ca60cd688f643ddefe0254cd17bec063b1c52d82c1b18c0e83a4d610601322ab"} Oct 30 22:04:26.006998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.006620 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca60cd688f643ddefe0254cd17bec063b1c52d82c1b18c0e83a4d610601322ab" Oct 30 22:04:26.009422 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.009391 4933 generic.go:356] "Generic (PLEG): container finished" podID="3705c57b-5255-41c4-be8a-5add2186e611" containerID="b690102398fefd437f472162939e582d26c95ddc3ebafebeb2cc62eb489cf55c" exitCode=0 Oct 30 22:04:26.009624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.009479 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" event={"ID":"3705c57b-5255-41c4-be8a-5add2186e611","Type":"ContainerDied","Data":"b690102398fefd437f472162939e582d26c95ddc3ebafebeb2cc62eb489cf55c"} Oct 30 22:04:26.211711 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.211687 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:26.272905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.272843 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-xks-ca\") pod \"fb08710c-0cab-423f-b093-4b22255fc6a7\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " Oct 30 22:04:26.272905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.272888 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-openshift-ca\") pod \"fb08710c-0cab-423f-b093-4b22255fc6a7\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " Oct 30 22:04:26.273096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.272926 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb08710c-0cab-423f-b093-4b22255fc6a7-kube-api-access-tg9mc\" (UniqueName: \"kubernetes.io/projected/fb08710c-0cab-423f-b093-4b22255fc6a7-kube-api-access-tg9mc\") pod \"fb08710c-0cab-423f-b093-4b22255fc6a7\" (UID: \"fb08710c-0cab-423f-b093-4b22255fc6a7\") " Oct 30 22:04:26.273262 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.273227 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-openshift-ca" pod "fb08710c-0cab-423f-b093-4b22255fc6a7" (UID: "fb08710c-0cab-423f-b093-4b22255fc6a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:26.273533 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.273514 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.275016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.274996 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-xks-ca" pod "fb08710c-0cab-423f-b093-4b22255fc6a7" (UID: "fb08710c-0cab-423f-b093-4b22255fc6a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:26.275198 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.275177 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb08710c-0cab-423f-b093-4b22255fc6a7-kube-api-access-tg9mc" pod "fb08710c-0cab-423f-b093-4b22255fc6a7" (UID: "fb08710c-0cab-423f-b093-4b22255fc6a7"). InnerVolumeSpecName "kube-api-access-tg9mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:26.374957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.374927 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb08710c-0cab-423f-b093-4b22255fc6a7-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:26.374957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:26.374953 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9mc\" (UniqueName: \"kubernetes.io/projected/fb08710c-0cab-423f-b093-4b22255fc6a7-kube-api-access-tg9mc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:27.018754 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.018728 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" Oct 30 22:04:27.019170 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.018766 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n" event={"ID":"fb08710c-0cab-423f-b093-4b22255fc6a7","Type":"ContainerDied","Data":"e2f03c1afb1ad474fac57dd8687b4959e78c9779eded2d3d5ede0f5023c0d153"} Oct 30 22:04:27.019170 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.018823 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2f03c1afb1ad474fac57dd8687b4959e78c9779eded2d3d5ede0f5023c0d153" Oct 30 22:04:27.193896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.193867 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:27.286927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.286904 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3705c57b-5255-41c4-be8a-5add2186e611-kube-api-access-xx552\" (UniqueName: \"kubernetes.io/projected/3705c57b-5255-41c4-be8a-5add2186e611-kube-api-access-xx552\") pod \"3705c57b-5255-41c4-be8a-5add2186e611\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " Oct 30 22:04:27.287075 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.287029 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-xks-ca\") pod \"3705c57b-5255-41c4-be8a-5add2186e611\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " Oct 30 22:04:27.287257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.287242 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-openshift-ca\") pod \"3705c57b-5255-41c4-be8a-5add2186e611\" (UID: \"3705c57b-5255-41c4-be8a-5add2186e611\") " Oct 30 22:04:27.287839 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.287662 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-openshift-ca" pod "3705c57b-5255-41c4-be8a-5add2186e611" (UID: "3705c57b-5255-41c4-be8a-5add2186e611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:27.287981 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.287880 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:27.289243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.289223 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-xks-ca" pod "3705c57b-5255-41c4-be8a-5add2186e611" (UID: "3705c57b-5255-41c4-be8a-5add2186e611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:27.289318 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.289248 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3705c57b-5255-41c4-be8a-5add2186e611-kube-api-access-xx552" pod "3705c57b-5255-41c4-be8a-5add2186e611" (UID: "3705c57b-5255-41c4-be8a-5add2186e611"). InnerVolumeSpecName "kube-api-access-xx552". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:27.389587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.389557 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx552\" (UniqueName: \"kubernetes.io/projected/3705c57b-5255-41c4-be8a-5add2186e611-kube-api-access-xx552\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:27.389587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:27.389582 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3705c57b-5255-41c4-be8a-5add2186e611-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:28.028669 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:28.028642 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" Oct 30 22:04:28.028669 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:28.028664 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp" event={"ID":"3705c57b-5255-41c4-be8a-5add2186e611","Type":"ContainerDied","Data":"9a3fd4ff89d9b09c98fa69a3897ecbbbc76b31bb08dfae36e8c5aabe53d3ad7a"} Oct 30 22:04:28.029203 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:28.028710 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a3fd4ff89d9b09c98fa69a3897ecbbbc76b31bb08dfae36e8c5aabe53d3ad7a" Oct 30 22:04:28.545175 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:04:28.545143 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:04:30.832404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.832365 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch"] Oct 30 22:04:30.833274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833255 4933 cpu_manager.go:414] "containerMap: removing container" podUID="cf9b4a49-9f70-4b43-b538-8ef7654eb444" containerName="revoke-keys" Oct 30 22:04:30.833312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833277 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9b4a49-9f70-4b43-b538-8ef7654eb444" containerName="revoke-keys" Oct 30 22:04:30.833312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833300 4933 cpu_manager.go:414] "containerMap: removing container" podUID="3705c57b-5255-41c4-be8a-5add2186e611" containerName="revoke-keys" Oct 30 22:04:30.833388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833310 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3705c57b-5255-41c4-be8a-5add2186e611" containerName="revoke-keys" Oct 30 22:04:30.833388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833349 4933 cpu_manager.go:414] "containerMap: removing container" podUID="fb08710c-0cab-423f-b093-4b22255fc6a7" containerName="revoke-keys" Oct 30 22:04:30.833388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833357 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb08710c-0cab-423f-b093-4b22255fc6a7" containerName="revoke-keys" Oct 30 22:04:30.833566 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833555 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9b4a49-9f70-4b43-b538-8ef7654eb444" containerName="revoke-keys" Oct 30 22:04:30.833606 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833570 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="3705c57b-5255-41c4-be8a-5add2186e611" containerName="revoke-keys" Oct 30 22:04:30.833606 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.833587 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb08710c-0cab-423f-b093-4b22255fc6a7" containerName="revoke-keys" Oct 30 22:04:30.837858 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.837842 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:30.893654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.893616 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch"] Oct 30 22:04:30.936389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.936341 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:30.936574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.936411 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:04:30.936574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.936427 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:04:30.936574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.936517 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:04:30.936574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.936532 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9sk\" (UniqueName: \"kubernetes.io/projected/11ffaecb-87c8-4a59-96ba-5820d4490e51-kube-api-access-2x9sk\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:30.936815 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.936714 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:30.941318 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:30.941302 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:04:31.038045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.038014 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9sk\" (UniqueName: \"kubernetes.io/projected/11ffaecb-87c8-4a59-96ba-5820d4490e51-kube-api-access-2x9sk\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:31.038180 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.038131 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:31.038269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.038251 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:31.038918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.038898 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:31.040789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.040768 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:31.096901 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.096826 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9sk\" (UniqueName: \"kubernetes.io/projected/11ffaecb-87c8-4a59-96ba-5820d4490e51-kube-api-access-2x9sk\") pod \"maas-api-revoke-sub-b9d3856f-v9sch\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:31.151919 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.151893 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:31.533081 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:31.533050 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ffaecb_87c8_4a59_96ba_5820d4490e51.slice/crio-106919aa38c7602b5e9d6e8afcdc877c359116aa7b54b5b51ad145c05c1f6880 WatchSource:0}: Error finding container 106919aa38c7602b5e9d6e8afcdc877c359116aa7b54b5b51ad145c05c1f6880: Status 404 returned error can't find the container with id 106919aa38c7602b5e9d6e8afcdc877c359116aa7b54b5b51ad145c05c1f6880 Oct 30 22:04:31.534752 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.534725 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch"] Oct 30 22:04:31.747868 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.747842 4933 scope.go:122] "RemoveContainer" containerID="1f0cb1d762b457ebc1c8f3150f204f18a3cac2ddd36318d8b7e44a820eff1bfe" Oct 30 22:04:31.758788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.758767 4933 scope.go:122] "RemoveContainer" containerID="a655fd2fccf85076bd47835c6632d953759f675121f40efed00520bb330992bc" Oct 30 22:04:31.768684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.768668 4933 scope.go:122] "RemoveContainer" containerID="ea8a07a584f5e9a44365b7e7937b58533da6be4e2090adcc45ed0afa2753b6db" Oct 30 22:04:31.777810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.777792 4933 scope.go:122] "RemoveContainer" containerID="673ae5df906a2c9e163deba50e6a1ec8581716f1447c2b0e60519de0967497fa" Oct 30 22:04:31.787139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.787121 4933 scope.go:122] "RemoveContainer" containerID="bf94123bd281eccf089930fcfe2d6a6de8fc720ce21f3bf4b88e2bbdb27b1e19" Oct 30 22:04:31.796604 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.796579 4933 scope.go:122] "RemoveContainer" containerID="765d7968ebdac621414903a1a4147ec341cf0806e3143c3aea2d4dae063a54f1" Oct 30 22:04:31.805915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:31.805894 4933 scope.go:122] "RemoveContainer" containerID="189cfecbf89b0d00670af04370856bab453964915aef500e4e6e5a8ba9a15102" Oct 30 22:04:32.071759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:32.071680 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" event={"ID":"11ffaecb-87c8-4a59-96ba-5820d4490e51","Type":"ContainerStarted","Data":"9ca6d219fa1c6949f1a400dcb0ecb4056f26cc7f147b36bd8a454faab9f91d1c"} Oct 30 22:04:32.071759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:32.071745 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" event={"ID":"11ffaecb-87c8-4a59-96ba-5820d4490e51","Type":"ContainerStarted","Data":"106919aa38c7602b5e9d6e8afcdc877c359116aa7b54b5b51ad145c05c1f6880"} Oct 30 22:04:32.189892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:32.189839 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" podStartSLOduration=2.189824797 podStartE2EDuration="2.189824797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:32.187855394 +0000 UTC m=+1320.869005287" watchObservedRunningTime="2026-10-30 22:04:32.189824797 +0000 UTC m=+1320.870974743" Oct 30 22:04:33.082822 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.082782 4933 generic.go:356] "Generic (PLEG): container finished" podID="11ffaecb-87c8-4a59-96ba-5820d4490e51" containerID="9ca6d219fa1c6949f1a400dcb0ecb4056f26cc7f147b36bd8a454faab9f91d1c" exitCode=0 Oct 30 22:04:33.083285 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.082842 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" event={"ID":"11ffaecb-87c8-4a59-96ba-5820d4490e51","Type":"ContainerDied","Data":"9ca6d219fa1c6949f1a400dcb0ecb4056f26cc7f147b36bd8a454faab9f91d1c"} Oct 30 22:04:33.355737 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.355658 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9"] Oct 30 22:04:33.360774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.360751 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.414473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.414446 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9"] Oct 30 22:04:33.467856 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.467823 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.468106 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.468074 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.468264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.468241 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4lh\" (UniqueName: \"kubernetes.io/projected/537c5c98-0834-4f21-b50c-d275c805eec3-kube-api-access-4b4lh\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.569581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.569542 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.569730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.569700 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4lh\" (UniqueName: \"kubernetes.io/projected/537c5c98-0834-4f21-b50c-d275c805eec3-kube-api-access-4b4lh\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.569873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.569854 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.570321 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.570292 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.572538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.572513 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.627922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.627858 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4lh\" (UniqueName: \"kubernetes.io/projected/537c5c98-0834-4f21-b50c-d275c805eec3-kube-api-access-4b4lh\") pod \"maas-api-revoke-sub-27f4cb5b-74xf9\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.676678 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.676653 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:33.940511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.940383 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r"] Oct 30 22:04:33.946804 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:33.946782 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.003269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.003244 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r"] Oct 30 22:04:34.055524 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.055498 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx"] Oct 30 22:04:34.062412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.062389 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39d9e865-r88fx"] Oct 30 22:04:34.068734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.068709 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9"] Oct 30 22:04:34.076687 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.076664 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.076850 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.076830 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.076889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.076864 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhx8v\" (UniqueName: \"kubernetes.io/projected/e60224e1-9943-4edb-bc33-132f7e58f663-kube-api-access-bhx8v\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.099826 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:34.099797 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537c5c98_0834_4f21_b50c_d275c805eec3.slice/crio-d4256c6b62062a2f1795f5a51814d5ee2bb60c8b738dee92f73faa38093a4c51 WatchSource:0}: Error finding container d4256c6b62062a2f1795f5a51814d5ee2bb60c8b738dee92f73faa38093a4c51: Status 404 returned error can't find the container with id d4256c6b62062a2f1795f5a51814d5ee2bb60c8b738dee92f73faa38093a4c51 Oct 30 22:04:34.178914 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.178889 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.179061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.178998 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.179061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.179021 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhx8v\" (UniqueName: \"kubernetes.io/projected/e60224e1-9943-4edb-bc33-132f7e58f663-kube-api-access-bhx8v\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.179637 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.179609 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.181453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.181432 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.237089 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.237030 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhx8v\" (UniqueName: \"kubernetes.io/projected/e60224e1-9943-4edb-bc33-132f7e58f663-kube-api-access-bhx8v\") pod \"maas-api-revoke-sub-38924ce4-jkh2r\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.257694 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.257674 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:34.261355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.261331 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:34.381199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.381153 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-openshift-ca\") pod \"11ffaecb-87c8-4a59-96ba-5820d4490e51\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " Oct 30 22:04:34.381365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.381234 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-xks-ca\") pod \"11ffaecb-87c8-4a59-96ba-5820d4490e51\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " Oct 30 22:04:34.381445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.381428 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ffaecb-87c8-4a59-96ba-5820d4490e51-kube-api-access-2x9sk\" (UniqueName: \"kubernetes.io/projected/11ffaecb-87c8-4a59-96ba-5820d4490e51-kube-api-access-2x9sk\") pod \"11ffaecb-87c8-4a59-96ba-5820d4490e51\" (UID: \"11ffaecb-87c8-4a59-96ba-5820d4490e51\") " Oct 30 22:04:34.381758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.381713 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-openshift-ca" pod "11ffaecb-87c8-4a59-96ba-5820d4490e51" (UID: "11ffaecb-87c8-4a59-96ba-5820d4490e51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:34.382556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.382531 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.384415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.384388 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ffaecb-87c8-4a59-96ba-5820d4490e51-kube-api-access-2x9sk" pod "11ffaecb-87c8-4a59-96ba-5820d4490e51" (UID: "11ffaecb-87c8-4a59-96ba-5820d4490e51"). InnerVolumeSpecName "kube-api-access-2x9sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:34.385031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.385008 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-xks-ca" pod "11ffaecb-87c8-4a59-96ba-5820d4490e51" (UID: "11ffaecb-87c8-4a59-96ba-5820d4490e51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:34.483774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.483747 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ffaecb-87c8-4a59-96ba-5820d4490e51-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.483774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.483772 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x9sk\" (UniqueName: \"kubernetes.io/projected/11ffaecb-87c8-4a59-96ba-5820d4490e51-kube-api-access-2x9sk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:34.651511 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:34.651462 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60224e1_9943_4edb_bc33_132f7e58f663.slice/crio-e7d301edc481f943ba16863046155d1965c3c7f529a2d4cf04a69b2077cce812 WatchSource:0}: Error finding container e7d301edc481f943ba16863046155d1965c3c7f529a2d4cf04a69b2077cce812: Status 404 returned error can't find the container with id e7d301edc481f943ba16863046155d1965c3c7f529a2d4cf04a69b2077cce812 Oct 30 22:04:34.652527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:34.652447 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r"] Oct 30 22:04:35.060997 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.060963 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp"] Oct 30 22:04:35.065474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.065443 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae607328-z62wp"] Oct 30 22:04:35.105563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.105426 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" event={"ID":"537c5c98-0834-4f21-b50c-d275c805eec3","Type":"ContainerStarted","Data":"df94b3eb0d01997180fb3e0acab63c84839f92d04fe9d693825052c909d6c1cb"} Oct 30 22:04:35.105563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.105520 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" event={"ID":"537c5c98-0834-4f21-b50c-d275c805eec3","Type":"ContainerStarted","Data":"d4256c6b62062a2f1795f5a51814d5ee2bb60c8b738dee92f73faa38093a4c51"} Oct 30 22:04:35.107409 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.107377 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" Oct 30 22:04:35.107571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.107431 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch" event={"ID":"11ffaecb-87c8-4a59-96ba-5820d4490e51","Type":"ContainerDied","Data":"106919aa38c7602b5e9d6e8afcdc877c359116aa7b54b5b51ad145c05c1f6880"} Oct 30 22:04:35.107571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.107470 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106919aa38c7602b5e9d6e8afcdc877c359116aa7b54b5b51ad145c05c1f6880" Oct 30 22:04:35.109952 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.109928 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" event={"ID":"e60224e1-9943-4edb-bc33-132f7e58f663","Type":"ContainerStarted","Data":"8940729a368601be36ee9646039fe79b0f6090edccb11562432ffa8d03d09a5e"} Oct 30 22:04:35.110059 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.109987 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" event={"ID":"e60224e1-9943-4edb-bc33-132f7e58f663","Type":"ContainerStarted","Data":"e7d301edc481f943ba16863046155d1965c3c7f529a2d4cf04a69b2077cce812"} Oct 30 22:04:35.242086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.242015 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" podStartSLOduration=2.241998305 podStartE2EDuration="2.241998305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:35.232654597 +0000 UTC m=+1323.913804490" watchObservedRunningTime="2026-10-30 22:04:35.241998305 +0000 UTC m=+1323.923148197" Oct 30 22:04:35.355090 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.354976 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" podStartSLOduration=2.354958822 podStartE2EDuration="2.354958822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:35.349138854 +0000 UTC m=+1324.030288748" watchObservedRunningTime="2026-10-30 22:04:35.354958822 +0000 UTC m=+1324.036108716" Oct 30 22:04:35.380916 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.380885 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n"] Oct 30 22:04:35.381100 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.381079 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" containerID="cri-o://55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49" gracePeriod=30 Oct 30 22:04:35.468274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.468241 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a14f46a-5ee8-484f-891e-ad7433c0196c" path="/var/lib/kubelet/pods/5a14f46a-5ee8-484f-891e-ad7433c0196c/volumes" Oct 30 22:04:35.468766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:35.468746 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efba10c2-e5d7-4ad0-9fe5-a7bf36639284" path="/var/lib/kubelet/pods/efba10c2-e5d7-4ad0-9fe5-a7bf36639284/volumes" Oct 30 22:04:36.120290 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:36.120259 4933 generic.go:356] "Generic (PLEG): container finished" podID="e60224e1-9943-4edb-bc33-132f7e58f663" containerID="8940729a368601be36ee9646039fe79b0f6090edccb11562432ffa8d03d09a5e" exitCode=0 Oct 30 22:04:36.120799 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:36.120350 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" event={"ID":"e60224e1-9943-4edb-bc33-132f7e58f663","Type":"ContainerDied","Data":"8940729a368601be36ee9646039fe79b0f6090edccb11562432ffa8d03d09a5e"} Oct 30 22:04:36.122261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:36.122235 4933 generic.go:356] "Generic (PLEG): container finished" podID="537c5c98-0834-4f21-b50c-d275c805eec3" containerID="df94b3eb0d01997180fb3e0acab63c84839f92d04fe9d693825052c909d6c1cb" exitCode=0 Oct 30 22:04:36.122376 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:36.122311 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" event={"ID":"537c5c98-0834-4f21-b50c-d275c805eec3","Type":"ContainerDied","Data":"df94b3eb0d01997180fb3e0acab63c84839f92d04fe9d693825052c909d6c1cb"} Oct 30 22:04:37.049336 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.049308 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk"] Oct 30 22:04:37.101784 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.101752 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d3be938-zxzxk"] Oct 30 22:04:37.331672 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.331652 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:37.335411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.335391 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:37.424709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.424682 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60224e1-9943-4edb-bc33-132f7e58f663-kube-api-access-bhx8v\" (UniqueName: \"kubernetes.io/projected/e60224e1-9943-4edb-bc33-132f7e58f663-kube-api-access-bhx8v\") pod \"e60224e1-9943-4edb-bc33-132f7e58f663\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " Oct 30 22:04:37.424858 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.424826 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-openshift-ca\") pod \"e60224e1-9943-4edb-bc33-132f7e58f663\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " Oct 30 22:04:37.424925 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.424904 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-xks-ca\") pod \"537c5c98-0834-4f21-b50c-d275c805eec3\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " Oct 30 22:04:37.424984 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.424946 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-xks-ca\") pod \"e60224e1-9943-4edb-bc33-132f7e58f663\" (UID: \"e60224e1-9943-4edb-bc33-132f7e58f663\") " Oct 30 22:04:37.425064 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.425035 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-openshift-ca\") pod \"537c5c98-0834-4f21-b50c-d275c805eec3\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " Oct 30 22:04:37.425156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.425141 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/537c5c98-0834-4f21-b50c-d275c805eec3-kube-api-access-4b4lh\" (UniqueName: \"kubernetes.io/projected/537c5c98-0834-4f21-b50c-d275c805eec3-kube-api-access-4b4lh\") pod \"537c5c98-0834-4f21-b50c-d275c805eec3\" (UID: \"537c5c98-0834-4f21-b50c-d275c805eec3\") " Oct 30 22:04:37.425225 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.425199 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-openshift-ca" pod "e60224e1-9943-4edb-bc33-132f7e58f663" (UID: "e60224e1-9943-4edb-bc33-132f7e58f663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:37.425421 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.425393 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-openshift-ca" pod "537c5c98-0834-4f21-b50c-d275c805eec3" (UID: "537c5c98-0834-4f21-b50c-d275c805eec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:37.426177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.426018 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.426177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.426045 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.427665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.427629 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537c5c98-0834-4f21-b50c-d275c805eec3-kube-api-access-4b4lh" pod "537c5c98-0834-4f21-b50c-d275c805eec3" (UID: "537c5c98-0834-4f21-b50c-d275c805eec3"). InnerVolumeSpecName "kube-api-access-4b4lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:37.427759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.427671 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60224e1-9943-4edb-bc33-132f7e58f663-kube-api-access-bhx8v" pod "e60224e1-9943-4edb-bc33-132f7e58f663" (UID: "e60224e1-9943-4edb-bc33-132f7e58f663"). InnerVolumeSpecName "kube-api-access-bhx8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:37.427759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.427716 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-xks-ca" pod "e60224e1-9943-4edb-bc33-132f7e58f663" (UID: "e60224e1-9943-4edb-bc33-132f7e58f663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:37.427759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.427741 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-xks-ca" pod "537c5c98-0834-4f21-b50c-d275c805eec3" (UID: "537c5c98-0834-4f21-b50c-d275c805eec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:37.453711 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.453690 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks"] Oct 30 22:04:37.454267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454255 4933 cpu_manager.go:414] "containerMap: removing container" podUID="537c5c98-0834-4f21-b50c-d275c805eec3" containerName="revoke-keys" Oct 30 22:04:37.454306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454269 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="537c5c98-0834-4f21-b50c-d275c805eec3" containerName="revoke-keys" Oct 30 22:04:37.454306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454283 4933 cpu_manager.go:414] "containerMap: removing container" podUID="11ffaecb-87c8-4a59-96ba-5820d4490e51" containerName="revoke-keys" Oct 30 22:04:37.454306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454288 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ffaecb-87c8-4a59-96ba-5820d4490e51" containerName="revoke-keys" Oct 30 22:04:37.454306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454302 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e60224e1-9943-4edb-bc33-132f7e58f663" containerName="revoke-keys" Oct 30 22:04:37.454412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454307 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60224e1-9943-4edb-bc33-132f7e58f663" containerName="revoke-keys" Oct 30 22:04:37.454412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454386 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ffaecb-87c8-4a59-96ba-5820d4490e51" containerName="revoke-keys" Oct 30 22:04:37.454412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454397 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60224e1-9943-4edb-bc33-132f7e58f663" containerName="revoke-keys" Oct 30 22:04:37.454412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.454408 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="537c5c98-0834-4f21-b50c-d275c805eec3" containerName="revoke-keys" Oct 30 22:04:37.458262 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.458246 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.469716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.469688 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da21d38b-237d-43d6-8b28-d2cde25debb1" path="/var/lib/kubelet/pods/da21d38b-237d-43d6-8b28-d2cde25debb1/volumes" Oct 30 22:04:37.517468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.517444 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks"] Oct 30 22:04:37.527410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.527384 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b4lh\" (UniqueName: \"kubernetes.io/projected/537c5c98-0834-4f21-b50c-d275c805eec3-kube-api-access-4b4lh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.527410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.527409 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhx8v\" (UniqueName: \"kubernetes.io/projected/e60224e1-9943-4edb-bc33-132f7e58f663-kube-api-access-bhx8v\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.527614 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.527418 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/537c5c98-0834-4f21-b50c-d275c805eec3-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.527614 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.527429 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60224e1-9943-4edb-bc33-132f7e58f663-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:37.635384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.635295 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.635384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.635331 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.635660 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.635563 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwhzq\" (UniqueName: \"kubernetes.io/projected/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-kube-api-access-xwhzq\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.737447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.737408 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.737652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.737455 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.737652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.737627 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwhzq\" (UniqueName: \"kubernetes.io/projected/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-kube-api-access-xwhzq\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.738361 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.738283 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.739982 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.739957 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:37.796963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:37.796934 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwhzq\" (UniqueName: \"kubernetes.io/projected/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-kube-api-access-xwhzq\") pod \"maas-api-revoke-sub-8a0d838f-lqkks\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:38.074287 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.074259 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:38.171676 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.171195 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" event={"ID":"537c5c98-0834-4f21-b50c-d275c805eec3","Type":"ContainerDied","Data":"d4256c6b62062a2f1795f5a51814d5ee2bb60c8b738dee92f73faa38093a4c51"} Oct 30 22:04:38.171676 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.171293 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4256c6b62062a2f1795f5a51814d5ee2bb60c8b738dee92f73faa38093a4c51" Oct 30 22:04:38.171676 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.171412 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9" Oct 30 22:04:38.182560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.182418 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" event={"ID":"e60224e1-9943-4edb-bc33-132f7e58f663","Type":"ContainerDied","Data":"e7d301edc481f943ba16863046155d1965c3c7f529a2d4cf04a69b2077cce812"} Oct 30 22:04:38.182560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.182470 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d301edc481f943ba16863046155d1965c3c7f529a2d4cf04a69b2077cce812" Oct 30 22:04:38.182805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.182612 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r" Oct 30 22:04:38.473464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.473436 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks"] Oct 30 22:04:38.821103 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.821069 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw"] Oct 30 22:04:38.826597 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.826575 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.875249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.875218 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvct\" (UniqueName: \"kubernetes.io/projected/edba9717-763b-4c13-81fe-ff9e0f0405be-kube-api-access-4vvct\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.875390 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.875334 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.875521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.875483 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.885111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.884844 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw"] Oct 30 22:04:38.976586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.976556 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvct\" (UniqueName: \"kubernetes.io/projected/edba9717-763b-4c13-81fe-ff9e0f0405be-kube-api-access-4vvct\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.976732 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.976611 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.976797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.976754 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.977345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.977322 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:38.979220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:38.979200 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:39.042095 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:39.042069 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvct\" (UniqueName: \"kubernetes.io/projected/edba9717-763b-4c13-81fe-ff9e0f0405be-kube-api-access-4vvct\") pod \"maas-api-revoke-sub-e20f7e45-2vjfw\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:39.142720 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:39.142653 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:39.195861 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:39.195830 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" event={"ID":"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4","Type":"ContainerStarted","Data":"e13779ed62f8304bdcbf548989a542bea3f243ad07f27da340ef40478a83f797"} Oct 30 22:04:39.196016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:39.195890 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" event={"ID":"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4","Type":"ContainerStarted","Data":"378aad99ad46a1bc3ed4dc71a71866ed4529616126253dd62d2bb1f2ce864b89"} Oct 30 22:04:39.315417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:39.315369 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" podStartSLOduration=2.315355459 podStartE2EDuration="2.315355459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:39.312866577 +0000 UTC m=+1327.994016471" watchObservedRunningTime="2026-10-30 22:04:39.315355459 +0000 UTC m=+1327.996505352" Oct 30 22:04:39.523012 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:39.522958 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedba9717_763b_4c13_81fe_ff9e0f0405be.slice/crio-5642b24cfb70303a284b162e514a4df77dc0ae1da246c5ed353514493ca0fa57 WatchSource:0}: Error finding container 5642b24cfb70303a284b162e514a4df77dc0ae1da246c5ed353514493ca0fa57: Status 404 returned error can't find the container with id 5642b24cfb70303a284b162e514a4df77dc0ae1da246c5ed353514493ca0fa57 Oct 30 22:04:39.524306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:39.524283 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw"] Oct 30 22:04:40.205747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.205646 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" event={"ID":"edba9717-763b-4c13-81fe-ff9e0f0405be","Type":"ContainerStarted","Data":"32dfb4ceaaf43d5cad7cf3dceafc41cd99f8fa1fc1696444d2268a061fa41c02"} Oct 30 22:04:40.205747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.205728 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" event={"ID":"edba9717-763b-4c13-81fe-ff9e0f0405be","Type":"ContainerStarted","Data":"5642b24cfb70303a284b162e514a4df77dc0ae1da246c5ed353514493ca0fa57"} Oct 30 22:04:40.208177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.208144 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" event={"ID":"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4","Type":"ContainerDied","Data":"e13779ed62f8304bdcbf548989a542bea3f243ad07f27da340ef40478a83f797"} Oct 30 22:04:40.208177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.208156 4933 generic.go:356] "Generic (PLEG): container finished" podID="5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" containerID="e13779ed62f8304bdcbf548989a542bea3f243ad07f27da340ef40478a83f797" exitCode=0 Oct 30 22:04:40.324065 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.323953 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" podStartSLOduration=2.323940806 podStartE2EDuration="2.323940806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:40.322008952 +0000 UTC m=+1329.003158847" watchObservedRunningTime="2026-10-30 22:04:40.323940806 +0000 UTC m=+1329.005090699" Oct 30 22:04:40.589756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.589727 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg"] Oct 30 22:04:40.595078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.595056 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.667132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.667097 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg"] Oct 30 22:04:40.702138 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.702103 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948kr\" (UniqueName: \"kubernetes.io/projected/6b356211-dcdc-4f92-b4a7-86c48e30eb15-kube-api-access-948kr\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.702262 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.702188 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.702324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.702274 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.705832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.705815 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 22:04:40.803298 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803273 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-data\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803432 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803329 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-certs\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803432 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803375 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-kube-api-access-8rmk7\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-kube-api-access-8rmk7\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803432 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803410 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-istio-token\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-istio-token\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803541 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-credential-socket\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803592 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c81900d6-5680-46cc-8815-387c4fa73013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81900d6-5680-46cc-8815-387c4fa73013-istio-podinfo\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803675 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803637 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-socket\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803675 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803654 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-data" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:40.803780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803683 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-certs" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:40.803856 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803825 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-envoy\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803867 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-credential-socket" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:40.803918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803892 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81900d6-5680-46cc-8815-387c4fa73013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81900d6-5680-46cc-8815-387c4fa73013-istiod-ca-cert\") pod \"c81900d6-5680-46cc-8815-387c4fa73013\" (UID: \"c81900d6-5680-46cc-8815-387c4fa73013\") " Oct 30 22:04:40.803918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.803897 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-socket" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:40.804203 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804149 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948kr\" (UniqueName: \"kubernetes.io/projected/6b356211-dcdc-4f92-b4a7-86c48e30eb15-kube-api-access-948kr\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.804400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804381 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.804664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804449 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.804664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804463 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81900d6-5680-46cc-8815-387c4fa73013-istiod-ca-cert" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:40.804790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804751 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.804790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804776 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.804895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804793 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81900d6-5680-46cc-8815-387c4fa73013-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.804895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804806 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.804895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.804822 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.805299 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.805253 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.806275 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.806255 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-kube-api-access-8rmk7" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "kube-api-access-8rmk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:40.806407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.806385 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-istio-token" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:40.806571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.806553 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81900d6-5680-46cc-8815-387c4fa73013-istio-podinfo" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:40.806624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.806571 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-envoy" pod "c81900d6-5680-46cc-8815-387c4fa73013" (UID: "c81900d6-5680-46cc-8815-387c4fa73013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:40.807644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.807626 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.863511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.863356 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-948kr\" (UniqueName: \"kubernetes.io/projected/6b356211-dcdc-4f92-b4a7-86c48e30eb15-kube-api-access-948kr\") pod \"maas-api-revoke-sub-4136dc62-n9zdg\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:40.907009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.906973 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rmk7\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-kube-api-access-8rmk7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.907009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.907012 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81900d6-5680-46cc-8815-387c4fa73013-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.907205 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.907026 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81900d6-5680-46cc-8815-387c4fa73013-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.907205 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.907040 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81900d6-5680-46cc-8815-387c4fa73013-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:40.912039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:40.912013 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:41.226297 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.226203 4933 generic.go:356] "Generic (PLEG): container finished" podID="edba9717-763b-4c13-81fe-ff9e0f0405be" containerID="32dfb4ceaaf43d5cad7cf3dceafc41cd99f8fa1fc1696444d2268a061fa41c02" exitCode=0 Oct 30 22:04:41.226297 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.226235 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" event={"ID":"edba9717-763b-4c13-81fe-ff9e0f0405be","Type":"ContainerDied","Data":"32dfb4ceaaf43d5cad7cf3dceafc41cd99f8fa1fc1696444d2268a061fa41c02"} Oct 30 22:04:41.227981 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.227960 4933 generic.go:356] "Generic (PLEG): container finished" podID="c81900d6-5680-46cc-8815-387c4fa73013" containerID="55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49" exitCode=0 Oct 30 22:04:41.228086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.228008 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" Oct 30 22:04:41.228086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.228036 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" event={"ID":"c81900d6-5680-46cc-8815-387c4fa73013","Type":"ContainerDied","Data":"55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49"} Oct 30 22:04:41.228086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.228081 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n" event={"ID":"c81900d6-5680-46cc-8815-387c4fa73013","Type":"ContainerDied","Data":"bb575b6c9b1a39f42ac717a3db4b988ce6bb9b53ffd04bbabde5885d957aca9f"} Oct 30 22:04:41.228176 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.228102 4933 scope.go:122] "RemoveContainer" containerID="55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49" Oct 30 22:04:41.240735 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.240714 4933 scope.go:122] "RemoveContainer" containerID="55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49" Oct 30 22:04:41.241055 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:04:41.241035 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49\": container with ID starting with 55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49 not found: ID does not exist" containerID="55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49" Oct 30 22:04:41.241142 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.241061 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49"} err="failed to get container status \"55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49\": rpc error: code = NotFound desc = could not find container \"55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49\": container with ID starting with 55627c0476eb83e9a9c02e5bcd4f52419a8a0f88d49a49138aad1373c4242f49 not found: ID does not exist" Oct 30 22:04:41.280529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.280500 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89"] Oct 30 22:04:41.281137 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.281124 4933 cpu_manager.go:414] "containerMap: removing container" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" Oct 30 22:04:41.281183 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.281138 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" Oct 30 22:04:41.281243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.281232 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81900d6-5680-46cc-8815-387c4fa73013" containerName="istio-proxy" Oct 30 22:04:41.286167 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.286147 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.316597 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.316548 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg"] Oct 30 22:04:41.321246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321222 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.321420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321402 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.321556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321471 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.321631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321575 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.321631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321619 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmmz\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-kube-api-access-thmmz\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.321889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321676 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.321889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321803 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.321889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321878 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf04c1d-4538-4574-aa8c-6b44e958e092-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.322040 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.321912 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.325126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.325025 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8"] Oct 30 22:04:41.364849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.355290 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.366537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.366458 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89"] Oct 30 22:04:41.420071 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.420050 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8"] Oct 30 22:04:41.425121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425095 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.425239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425202 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425253 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.425305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425291 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b949074-23f2-4c8b-bd61-d5231364de34-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.425424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425350 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425368 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425396 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.425602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425437 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425467 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmmz\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-kube-api-access-thmmz\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425530 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425563 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpkk\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-kube-api-access-xkpkk\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.425810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425619 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425648 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b949074-23f2-4c8b-bd61-d5231364de34-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.425810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425720 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf04c1d-4538-4574-aa8c-6b44e958e092-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425752 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.425810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425781 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.426056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425829 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.426056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425868 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.426056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.425942 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.426244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.426213 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.426393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.426369 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.426636 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.426615 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.428335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.428228 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf04c1d-4538-4574-aa8c-6b44e958e092-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.436533 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.431164 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.447810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.447784 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.485285 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.485215 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n"] Oct 30 22:04:41.490596 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.490574 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-dlj2n"] Oct 30 22:04:41.527164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527133 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.527280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527168 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.527280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527193 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.527280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527240 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.527690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527665 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.527860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527718 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b949074-23f2-4c8b-bd61-d5231364de34-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.527860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527818 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-data\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.527860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527829 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.528026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.527879 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-certs\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.528026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.528001 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpkk\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-kube-api-access-xkpkk\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.528139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.528071 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b949074-23f2-4c8b-bd61-d5231364de34-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.528778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.528756 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b949074-23f2-4c8b-bd61-d5231364de34-istiod-ca-cert\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.528895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.528844 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.528968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.528928 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-credential-socket\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.530901 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.530880 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-envoy\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.531471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.531443 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmmz\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-kube-api-access-thmmz\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.531650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.531626 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b949074-23f2-4c8b-bd61-d5231364de34-istio-podinfo\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.531705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.531676 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.587654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.587630 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-istio-token\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.587755 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.587701 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpkk\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-kube-api-access-xkpkk\") pod \"e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.587991 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.587972 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:41.597535 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.597513 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz"] Oct 30 22:04:41.598293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.598273 4933 cpu_manager.go:414] "containerMap: removing container" podUID="5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" containerName="revoke-keys" Oct 30 22:04:41.598377 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.598295 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" containerName="revoke-keys" Oct 30 22:04:41.598652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.598537 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" containerName="revoke-keys" Oct 30 22:04:41.602450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.602434 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.602584 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.602523 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:41.625450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.625430 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf"] Oct 30 22:04:41.629468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.629049 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-kube-api-access-xwhzq\" (UniqueName: \"kubernetes.io/projected/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-kube-api-access-xwhzq\") pod \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " Oct 30 22:04:41.629468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.629146 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-xks-ca\") pod \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " Oct 30 22:04:41.629468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.629362 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-openshift-ca\") pod \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\" (UID: \"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4\") " Oct 30 22:04:41.630010 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.629986 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-openshift-ca" pod "5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" (UID: "5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:41.632759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.632276 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.634452 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.634416 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-kube-api-access-xwhzq" pod "5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" (UID: "5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4"). InnerVolumeSpecName "kube-api-access-xwhzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:41.634452 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.634418 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-xks-ca" pod "5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" (UID: "5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:41.663267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.663238 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz"] Oct 30 22:04:41.677574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.677419 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:41.699898 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.699725 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf"] Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731544 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731605 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731636 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731687 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731722 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731751 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzn6b\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-kube-api-access-hzn6b\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731814 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731883 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.731917 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732030 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732056 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732090 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732141 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732197 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732240 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732265 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732296 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7d7\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-kube-api-access-ml7d7\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732320 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732388 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732405 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwhzq\" (UniqueName: \"kubernetes.io/projected/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-kube-api-access-xwhzq\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:41.732661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.732419 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:41.835801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.835087 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.835801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.835137 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.835801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.835171 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.835801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.835700 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840684 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840767 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840818 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840843 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840873 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7d7\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-kube-api-access-ml7d7\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840898 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840935 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.840984 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.841016 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.841151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.841081 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.842931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.841820 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.842931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.841879 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzn6b\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-kube-api-access-hzn6b\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.842931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.841964 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.842931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.842061 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.842931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.842102 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.842931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.842612 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-data\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.844567 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.845713 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.845957 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.846513 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istiod-ca-cert\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.846612 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-certs\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.846896 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.848744 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.849394 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.854017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.853909 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.854458 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.854289 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-envoy\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.854595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.854574 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-credential-socket\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.866992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.866968 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-podinfo\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.899985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.899958 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.903829 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.903801 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzn6b\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-kube-api-access-hzn6b\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:41.903975 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.903957 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7d7\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-kube-api-access-ml7d7\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.906502 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.906466 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-token\") pod \"e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.920690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.920669 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:41.950312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:41.950288 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:42.020281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.020257 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89"] Oct 30 22:04:42.024855 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:42.024824 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf04c1d_4538_4574_aa8c_6b44e958e092.slice/crio-f94a3427b701ddaffb90f16a1add66ece24e70879accfae1013b0693d9c0ddf3 WatchSource:0}: Error finding container f94a3427b701ddaffb90f16a1add66ece24e70879accfae1013b0693d9c0ddf3: Status 404 returned error can't find the container with id f94a3427b701ddaffb90f16a1add66ece24e70879accfae1013b0693d9c0ddf3 Oct 30 22:04:42.027245 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.027178 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.027335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.027258 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.027335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.027307 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.104145 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:42.104080 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b949074_23f2_4c8b_bd61_d5231364de34.slice/crio-8e5682638c238eb100ff59ca071873fa8ec02f44a3eb01abb943b889ab9c2151 WatchSource:0}: Error finding container 8e5682638c238eb100ff59ca071873fa8ec02f44a3eb01abb943b889ab9c2151: Status 404 returned error can't find the container with id 8e5682638c238eb100ff59ca071873fa8ec02f44a3eb01abb943b889ab9c2151 Oct 30 22:04:42.106880 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.106704 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8"] Oct 30 22:04:42.107456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.107425 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.107579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.107531 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.107579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.107562 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.247333 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.247291 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" event={"ID":"dcf04c1d-4538-4574-aa8c-6b44e958e092","Type":"ContainerStarted","Data":"5e2aa04f2501f4d8a9037f44a91c195aa9f71392f09dedf9dd3a6f38e5793678"} Oct 30 22:04:42.247530 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.247398 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" event={"ID":"dcf04c1d-4538-4574-aa8c-6b44e958e092","Type":"ContainerStarted","Data":"f94a3427b701ddaffb90f16a1add66ece24e70879accfae1013b0693d9c0ddf3"} Oct 30 22:04:42.253067 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.253000 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" event={"ID":"5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4","Type":"ContainerDied","Data":"378aad99ad46a1bc3ed4dc71a71866ed4529616126253dd62d2bb1f2ce864b89"} Oct 30 22:04:42.253328 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.253313 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378aad99ad46a1bc3ed4dc71a71866ed4529616126253dd62d2bb1f2ce864b89" Oct 30 22:04:42.253449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.253261 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks" Oct 30 22:04:42.263767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.263742 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" event={"ID":"4b949074-23f2-4c8b-bd61-d5231364de34","Type":"ContainerStarted","Data":"d4f5ec9a84413291bb4bccd02755d49f980b41780fac98ad6495f0878373fb05"} Oct 30 22:04:42.264646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.264624 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" event={"ID":"4b949074-23f2-4c8b-bd61-d5231364de34","Type":"ContainerStarted","Data":"8e5682638c238eb100ff59ca071873fa8ec02f44a3eb01abb943b889ab9c2151"} Oct 30 22:04:42.267632 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.267610 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" event={"ID":"6b356211-dcdc-4f92-b4a7-86c48e30eb15","Type":"ContainerStarted","Data":"cc0c68f3fd3e3eabeea8090d64f8d5afd11b8e33525b574c64676129c5890e29"} Oct 30 22:04:42.270182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.270161 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" event={"ID":"6b356211-dcdc-4f92-b4a7-86c48e30eb15","Type":"ContainerStarted","Data":"19262122bc09557c1a5b638c383959aa6e3d9d8b07b9a29b87e6f91b712d5324"} Oct 30 22:04:42.352814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.341302 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz"] Oct 30 22:04:42.352814 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:42.342001 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d6876_36a7_4faa_92a3_3ddc9bd511a1.slice/crio-f442450e5e169de10a99b89c874667d5f1ba7d20b63d46830eaf33530b1d9697 WatchSource:0}: Error finding container f442450e5e169de10a99b89c874667d5f1ba7d20b63d46830eaf33530b1d9697: Status 404 returned error can't find the container with id f442450e5e169de10a99b89c874667d5f1ba7d20b63d46830eaf33530b1d9697 Oct 30 22:04:42.352814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.347091 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.352814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.347182 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.352814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.347223 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.372785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.372406 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podStartSLOduration=1.372386886 podStartE2EDuration="1.372386886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.37047856 +0000 UTC m=+1331.051628465" watchObservedRunningTime="2026-10-30 22:04:42.372386886 +0000 UTC m=+1331.053536780" Oct 30 22:04:42.406031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.397059 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf"] Oct 30 22:04:42.412556 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:04:42.412521 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9531138b_2ae4_4de4_a05c_b88b2d69f20c.slice/crio-66fb2bd463784215d7298e2047dd8a8d44a8d544a41c0e5ca567bb3fb6152295 WatchSource:0}: Error finding container 66fb2bd463784215d7298e2047dd8a8d44a8d544a41c0e5ca567bb3fb6152295: Status 404 returned error can't find the container with id 66fb2bd463784215d7298e2047dd8a8d44a8d544a41c0e5ca567bb3fb6152295 Oct 30 22:04:42.433177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.433109 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.433306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.433224 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.433306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.433270 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:04:42.603203 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.603177 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:42.661781 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.661748 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:42.679693 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.679645 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:42.680002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.679960 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" podStartSLOduration=2.67994848 podStartE2EDuration="2.67994848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.502775972 +0000 UTC m=+1331.183925866" watchObservedRunningTime="2026-10-30 22:04:42.67994848 +0000 UTC m=+1331.361098398" Oct 30 22:04:42.686430 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.686318 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podStartSLOduration=1.6863000989999999 podStartE2EDuration="1.686300099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:42.679346418 +0000 UTC m=+1331.360496327" watchObservedRunningTime="2026-10-30 22:04:42.686300099 +0000 UTC m=+1331.367449993" Oct 30 22:04:42.764327 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.764293 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edba9717-763b-4c13-81fe-ff9e0f0405be-kube-api-access-4vvct\" (UniqueName: \"kubernetes.io/projected/edba9717-763b-4c13-81fe-ff9e0f0405be-kube-api-access-4vvct\") pod \"edba9717-763b-4c13-81fe-ff9e0f0405be\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " Oct 30 22:04:42.766616 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.766587 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-xks-ca\") pod \"edba9717-763b-4c13-81fe-ff9e0f0405be\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " Oct 30 22:04:42.766870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.766854 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-openshift-ca\") pod \"edba9717-763b-4c13-81fe-ff9e0f0405be\" (UID: \"edba9717-763b-4c13-81fe-ff9e0f0405be\") " Oct 30 22:04:42.768166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.768142 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-openshift-ca" pod "edba9717-763b-4c13-81fe-ff9e0f0405be" (UID: "edba9717-763b-4c13-81fe-ff9e0f0405be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:42.775930 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.775896 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edba9717-763b-4c13-81fe-ff9e0f0405be-kube-api-access-4vvct" pod "edba9717-763b-4c13-81fe-ff9e0f0405be" (UID: "edba9717-763b-4c13-81fe-ff9e0f0405be"). InnerVolumeSpecName "kube-api-access-4vvct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:42.780626 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.780601 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-xks-ca" pod "edba9717-763b-4c13-81fe-ff9e0f0405be" (UID: "edba9717-763b-4c13-81fe-ff9e0f0405be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:42.870070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.869962 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:42.870274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.870256 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edba9717-763b-4c13-81fe-ff9e0f0405be-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:42.870358 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:42.870347 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvct\" (UniqueName: \"kubernetes.io/projected/edba9717-763b-4c13-81fe-ff9e0f0405be-kube-api-access-4vvct\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:43.313620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.313586 4933 generic.go:356] "Generic (PLEG): container finished" podID="6b356211-dcdc-4f92-b4a7-86c48e30eb15" containerID="cc0c68f3fd3e3eabeea8090d64f8d5afd11b8e33525b574c64676129c5890e29" exitCode=0 Oct 30 22:04:43.313823 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.313664 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" event={"ID":"6b356211-dcdc-4f92-b4a7-86c48e30eb15","Type":"ContainerDied","Data":"cc0c68f3fd3e3eabeea8090d64f8d5afd11b8e33525b574c64676129c5890e29"} Oct 30 22:04:43.315721 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.315696 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" event={"ID":"9531138b-2ae4-4de4-a05c-b88b2d69f20c","Type":"ContainerStarted","Data":"237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1"} Oct 30 22:04:43.315855 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.315733 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" event={"ID":"9531138b-2ae4-4de4-a05c-b88b2d69f20c","Type":"ContainerStarted","Data":"66fb2bd463784215d7298e2047dd8a8d44a8d544a41c0e5ca567bb3fb6152295"} Oct 30 22:04:43.317302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.317284 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" Oct 30 22:04:43.317454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.317439 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw" event={"ID":"edba9717-763b-4c13-81fe-ff9e0f0405be","Type":"ContainerDied","Data":"5642b24cfb70303a284b162e514a4df77dc0ae1da246c5ed353514493ca0fa57"} Oct 30 22:04:43.317569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.317559 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5642b24cfb70303a284b162e514a4df77dc0ae1da246c5ed353514493ca0fa57" Oct 30 22:04:43.319717 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.319700 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" event={"ID":"405d6876-36a7-4faa-92a3-3ddc9bd511a1","Type":"ContainerStarted","Data":"741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f"} Oct 30 22:04:43.319860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.319844 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" event={"ID":"405d6876-36a7-4faa-92a3-3ddc9bd511a1","Type":"ContainerStarted","Data":"f442450e5e169de10a99b89c874667d5f1ba7d20b63d46830eaf33530b1d9697"} Oct 30 22:04:43.519640 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.519572 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81900d6-5680-46cc-8815-387c4fa73013" path="/var/lib/kubelet/pods/c81900d6-5680-46cc-8815-387c4fa73013/volumes" Oct 30 22:04:43.684578 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.681673 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:43.684578 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.681733 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:43.691572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.691230 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podStartSLOduration=2.6912082440000002 podStartE2EDuration="2.691208244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:43.672006307 +0000 UTC m=+1332.353156201" watchObservedRunningTime="2026-10-30 22:04:43.691208244 +0000 UTC m=+1332.372358138" Oct 30 22:04:43.692283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.692259 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:43.692506 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.692464 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:43.943841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.941826 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:43.958254 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.958202 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:43.958572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.958541 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:43.960446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.960425 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:43.971714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.971666 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:43.972015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:43.971969 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:44.631518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.616259 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:44.633733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.633657 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:44.634265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.631814 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:44.651779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.651744 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:44.685941 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.685903 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:44.686530 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.686480 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:44.691103 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.691072 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:44.732647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.732597 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:44.734932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.732897 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:44.789252 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.788781 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:44.823190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.823124 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podStartSLOduration=3.8231031680000003 podStartE2EDuration="3.823103168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:43.813314359 +0000 UTC m=+1332.494464264" watchObservedRunningTime="2026-10-30 22:04:44.823103168 +0000 UTC m=+1333.504253061" Oct 30 22:04:44.946954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.946866 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:44.947200 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.947172 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:44.972771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.963629 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:44.972771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.963689 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:44.972771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.964295 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p"] Oct 30 22:04:44.982874 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.973192 4933 cpu_manager.go:414] "containerMap: removing container" podUID="edba9717-763b-4c13-81fe-ff9e0f0405be" containerName="revoke-keys" Oct 30 22:04:44.982874 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.973216 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edba9717-763b-4c13-81fe-ff9e0f0405be" containerName="revoke-keys" Oct 30 22:04:45.012542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:44.973479 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="edba9717-763b-4c13-81fe-ff9e0f0405be" containerName="revoke-keys" Oct 30 22:04:45.012542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.004691 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:45.021843 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.021808 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:45.074534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.067287 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:45.729702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.729652 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:45.794536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.760234 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:45.794900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.794868 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:45.795035 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.795020 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gslbk\" (UniqueName: \"kubernetes.io/projected/9edbffec-25c8-483f-aa6a-259737a28b10-kube-api-access-gslbk\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:45.844515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.810996 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:45.844515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.818760 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:45.844515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.827618 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.891244 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.891532 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.891733 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.892006 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.892082 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.892144 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.893946 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.899035 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.899136 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.899190 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.900067 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.900112 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.903325 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.903422 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:45.908547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.903461 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gslbk\" (UniqueName: \"kubernetes.io/projected/9edbffec-25c8-483f-aa6a-259737a28b10-kube-api-access-gslbk\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:45.968435 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.949861 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:45.968435 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.949931 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:45.968435 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:45.968051 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:46.342515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.292099 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:04:46.476572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.439800 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:46.476572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.463597 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:46.476572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.463670 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:46.485739 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.485706 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gslbk\" (UniqueName: \"kubernetes.io/projected/9edbffec-25c8-483f-aa6a-259737a28b10-kube-api-access-gslbk\") pod \"maas-api-revoke-sub-e3a35d24-pp75p\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:46.529525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.522991 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.529525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.523047 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:46.529525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.523119 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.529525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.523139 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:46.539590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.538835 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:04:46.539590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.538882 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p"] Oct 30 22:04:46.539590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.539171 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:04:46.539590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.539241 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:04:46.539590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.539297 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:04:46.611700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.611669 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.611963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.611938 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:46.623678 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.623651 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:46.656519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.656465 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:46.695641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.695594 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b356211-dcdc-4f92-b4a7-86c48e30eb15-kube-api-access-948kr\" (UniqueName: \"kubernetes.io/projected/6b356211-dcdc-4f92-b4a7-86c48e30eb15-kube-api-access-948kr\") pod \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " Oct 30 22:04:46.709719 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.709692 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-openshift-ca\") pod \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " Oct 30 22:04:46.710367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.710346 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-xks-ca\") pod \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\" (UID: \"6b356211-dcdc-4f92-b4a7-86c48e30eb15\") " Oct 30 22:04:46.715811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.710221 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-openshift-ca" pod "6b356211-dcdc-4f92-b4a7-86c48e30eb15" (UID: "6b356211-dcdc-4f92-b4a7-86c48e30eb15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:46.821517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.807378 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b356211-dcdc-4f92-b4a7-86c48e30eb15-kube-api-access-948kr" pod "6b356211-dcdc-4f92-b4a7-86c48e30eb15" (UID: "6b356211-dcdc-4f92-b4a7-86c48e30eb15"). InnerVolumeSpecName "kube-api-access-948kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:46.852923 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.851879 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:46.852923 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.852279 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:04:46.914575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.854470 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-948kr\" (UniqueName: \"kubernetes.io/projected/6b356211-dcdc-4f92-b4a7-86c48e30eb15-kube-api-access-948kr\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:46.914575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.905460 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:46.914575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.894939 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:46.914575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.894994 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:46.914575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.895013 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:46.914575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.894974 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:46.914575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.906685 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:04:46.939518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.933717 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.939518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.933791 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:46.992554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.984055 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.992554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.984108 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:46.992554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.986710 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:46.992554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:46.986769 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:47.016011 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.015985 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:04:47.086745 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.083659 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-xks-ca" pod "6b356211-dcdc-4f92-b4a7-86c48e30eb15" (UID: "6b356211-dcdc-4f92-b4a7-86c48e30eb15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:47.235433 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.150894 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b356211-dcdc-4f92-b4a7-86c48e30eb15-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:47.547692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.490714 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:47.547692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.495668 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:47.547692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.553255 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:47.586371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.586325 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:47.586780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.586752 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 22:04:47.625307 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.625264 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 30 22:04:47.655801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.655762 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:47.657181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.657139 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:47.705360 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.705271 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:47.705615 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.705585 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:47.706425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.706398 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:47.708217 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.708193 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Oct 30 22:04:47.710797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.710774 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:47.710953 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.710932 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:47.886287 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.886244 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.111318 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.903335 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" Oct 30 22:04:48.127021 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.904597 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.129321 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:47.960608 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:48.130609 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.056967 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.151541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.057057 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:48.151541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.135105 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:48.151541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.057236 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:48.151541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.057209 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:48.151541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.093565 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:48.151541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.136396 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:48.151541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.057255 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:04:48.513475 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:04:48.511871 4933 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.044s" Oct 30 22:04:48.514661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.514620 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.523515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.511910 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg" event={"ID":"6b356211-dcdc-4f92-b4a7-86c48e30eb15","Type":"ContainerDied","Data":"19262122bc09557c1a5b638c383959aa6e3d9d8b07b9a29b87e6f91b712d5324"} Oct 30 22:04:48.523515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.519614 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19262122bc09557c1a5b638c383959aa6e3d9d8b07b9a29b87e6f91b712d5324" Oct 30 22:04:48.523515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.519800 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:04:48.523515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.519850 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:04:48.523515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.519889 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:04:48.523515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.519925 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:04:48.574573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.551378 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:04:48.663527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.553551 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.677519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.665365 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.698477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.696280 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:48.698477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.696338 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:48.698477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.697432 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:48.698477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.697503 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:48.698477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.697725 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.698477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.697788 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:48.698477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.698065 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:48.722268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.722197 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:48.722897 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:48.722877 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:04:49.061183 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.061147 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p"] Oct 30 22:04:49.134847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.134800 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:49.160516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.135344 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:49.178585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.141539 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:49.179902 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.142773 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:49.179902 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.142806 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:04:49.180091 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.149562 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:49.180154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.180121 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:49.180201 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.160149 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:49.180201 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.180183 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:49.571522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.570911 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 30 22:04:49.676515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.643448 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:49.677134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.677099 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:49.768101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.768062 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:49.785540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.779940 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:04:49.785540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.781619 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:49.786318 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.786294 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:49.793786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.793735 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:49.793989 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.788387 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:49.870381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.814072 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:49.904514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:49.788269 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:50.179066 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.179025 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:50.182670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.182632 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.187624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.187577 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.243802 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:04:50.243764 4933 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b356211_dcdc_4f92_b4a7_86c48e30eb15.slice/crio-19262122bc09557c1a5b638c383959aa6e3d9d8b07b9a29b87e6f91b712d5324\": RecentStats: unable to find data in memory cache]" Oct 30 22:04:50.285581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.285542 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 22:04:50.287947 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.287921 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:50.304531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.304477 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:50.304849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.304829 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:50.304998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.304977 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:50.305154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.305139 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:50.305280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.305262 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:50.350518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.338386 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:50.368844 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.368730 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.704396 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.704297 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:50.708600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.708556 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:50.734516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.733841 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:50.734516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.733903 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:50.734516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.733999 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.759514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.747770 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:04:50.808650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.808604 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:50.809688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.809656 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:50.938516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.829989 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 22:04:50.940273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.830022 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:04:50.940878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.830041 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:04:50.967570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.942082 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:51.060840 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.060801 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:51.061047 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:04:51.055185 4933 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.171s" Oct 30 22:04:51.061171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.061155 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" event={"ID":"9edbffec-25c8-483f-aa6a-259737a28b10","Type":"ContainerStarted","Data":"469c435ce67b68fca35e22f32c29ab4b97bf2bfcc3982079caef58735c5a7d6c"} Oct 30 22:04:51.061410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.061397 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:04:51.077516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:50.969105 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:51.422228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.416573 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:51.422228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.123155 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:51.422228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.185175 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.422228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.185215 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:51.422228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.203945 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:51.422228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.301768 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:51.422228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.388754 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.543875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.543842 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz"] Oct 30 22:04:51.689623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.645556 4933 cpu_manager.go:414] "containerMap: removing container" podUID="6b356211-dcdc-4f92-b4a7-86c48e30eb15" containerName="revoke-keys" Oct 30 22:04:51.693511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.689945 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b356211-dcdc-4f92-b4a7-86c48e30eb15" containerName="revoke-keys" Oct 30 22:04:51.718076 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.693847 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:51.718076 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.693904 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:51.757576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.590832 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:51.758212 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.758186 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b356211-dcdc-4f92-b4a7-86c48e30eb15" containerName="revoke-keys" Oct 30 22:04:51.797729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.773618 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.797729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.773931 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:51.848516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.839605 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:51.848516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.839674 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:51.888654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.888608 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.889255 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.889226 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:51.954692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:51.954573 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:04:52.046520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.021781 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:04:52.047447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.035888 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:52.047690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.047659 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.047833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.036989 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.114251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.114217 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:52.127627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.127578 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:52.187513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.163627 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqmwf\" (UniqueName: \"kubernetes.io/projected/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-kube-api-access-nqmwf\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.220570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.205942 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.220570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.206011 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.346518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.340223 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqmwf\" (UniqueName: \"kubernetes.io/projected/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-kube-api-access-nqmwf\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.346518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.340355 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.346518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.340405 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.380569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.354455 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.421841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.421541 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.421841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.421641 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:52.421841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.421680 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:04:52.421841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.421720 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:52.421841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.421756 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:52.421841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.421792 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:52.446516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.441684 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz"] Oct 30 22:04:52.638605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.472776 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.638605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.545300 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:52.638605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.630267 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:52.638605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.592796 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.638605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.546555 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.638605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.632865 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqmwf\" (UniqueName: \"kubernetes.io/projected/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-kube-api-access-nqmwf\") pod \"maas-api-revoke-sub-b0cda51d-sclgz\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:52.644201 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.644180 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974"] Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.862590 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76f898c4-9n974"] Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.760445 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.719991 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.863015 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.722289 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.863066 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.821842 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.872516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.821878 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:04:52.968595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.949035 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:52.968595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.957681 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:04:52.968595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.958718 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:52.968595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.958754 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:52.968595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.958819 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:52.968595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.959857 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:52.968595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:52.959892 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:53.129713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.129677 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:04:53.154302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.154031 4933 generic.go:356] "Generic (PLEG): container finished" podID="9edbffec-25c8-483f-aa6a-259737a28b10" containerID="a2901e5fd1d1c58578ebe8f848401f7052741c2de51e576170da2a6c80dff1da" exitCode=0 Oct 30 22:04:53.154302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.154209 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" event={"ID":"9edbffec-25c8-483f-aa6a-259737a28b10","Type":"ContainerDied","Data":"a2901e5fd1d1c58578ebe8f848401f7052741c2de51e576170da2a6c80dff1da"} Oct 30 22:04:53.733115 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.732988 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:53.741798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.741757 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:53.755079 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.755025 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:53.830517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.755643 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:04:53.830517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.787867 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:53.830517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.778652 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:53.830517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.788604 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:04:53.830517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.800093 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:53.830517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.800345 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:53.854870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.853222 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 30 22:04:53.854870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.854252 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:53.854870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.854309 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:53.914512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:53.899168 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.028618 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.022588 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:54.049540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.048602 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.050052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.049892 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:04:54.113825 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.113780 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:04:54.116523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.116129 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:54.116523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.116181 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:54.129515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.118747 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:54.189520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.185159 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:54.189520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.185244 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:54.567776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.567737 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e6546e-fc61-47ed-abb7-613e4e2e79ab" path="/var/lib/kubelet/pods/88e6546e-fc61-47ed-abb7-613e4e2e79ab/volumes" Oct 30 22:04:54.574569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.568529 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp"] Oct 30 22:04:54.606483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.606451 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp"] Oct 30 22:04:54.639058 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.606785 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:54.648642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.648611 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:54.648859 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.648830 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:54.684638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.681633 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:54.684638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.681690 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:54.698232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.698198 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjzf\" (UniqueName: \"kubernetes.io/projected/79ded627-4bad-4d13-96a2-fef1fb1b7452-kube-api-access-qwjzf\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:54.856604 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.856567 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:54.857270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.857241 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:54.926520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.899422 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.926520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.790650 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:54.926520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.790688 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.926520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.790710 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:54.926520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.790727 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:04:54.926520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.790747 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.957142 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.745628 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:54.958159 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.739968 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.994036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.959467 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:54.994539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.994481 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:54.994770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.790610 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:54.998509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.967054 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:54.998683 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:54.998651 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:55.079576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.034984 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:55.363617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.363435 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:55.369948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.369913 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:04:55.370371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.370344 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:04:55.370718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.370695 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:04:55.562139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.562101 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:55.562538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.562515 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:55.562847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.562830 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjzf\" (UniqueName: \"kubernetes.io/projected/79ded627-4bad-4d13-96a2-fef1fb1b7452-kube-api-access-qwjzf\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:55.708288 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.564872 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:55.708559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.610023 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:55.708734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.708704 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.901304 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.948187 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.959359 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.959413 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.962630 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.967432 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.969917 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.975746 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.977677 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.996707 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.996754 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:56.014617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:55.996818 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.103573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.059998 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.200517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.127800 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:04:56.324516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.324199 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjzf\" (UniqueName: \"kubernetes.io/projected/79ded627-4bad-4d13-96a2-fef1fb1b7452-kube-api-access-qwjzf\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:56.346516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.340114 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.375515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.374726 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.375515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.375446 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.404698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.376041 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:04:56.404698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.377568 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:04:56.428273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.428240 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.428581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.428552 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:56.450512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.450461 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz"] Oct 30 22:04:56.616024 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.615986 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:04:56.627835 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.627807 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:04:56.629155 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.629132 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.629324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.629301 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:56.709511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.687763 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:56.727535 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.695940 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.727753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.727728 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 22:04:56.946569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.927172 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.965795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.965759 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 22:04:56.966003 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.957745 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:04:56.966344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.957711 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.966954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.965235 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:56.967100 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.967076 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:56.967234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.964746 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:04:56.967591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.964989 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:56.967790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.967776 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:56.983788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.972796 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:04:56.983973 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.976570 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.984461 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.978068 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:56.984856 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.979409 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:04:56.985438 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:56.985361 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:57.029519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.027090 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edbffec-25c8-483f-aa6a-259737a28b10-kube-api-access-gslbk\" (UniqueName: \"kubernetes.io/projected/9edbffec-25c8-483f-aa6a-259737a28b10-kube-api-access-gslbk\") pod \"9edbffec-25c8-483f-aa6a-259737a28b10\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " Oct 30 22:04:57.029519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.027253 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-openshift-ca\") pod \"9edbffec-25c8-483f-aa6a-259737a28b10\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " Oct 30 22:04:57.029519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.027393 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-xks-ca\") pod \"9edbffec-25c8-483f-aa6a-259737a28b10\" (UID: \"9edbffec-25c8-483f-aa6a-259737a28b10\") " Oct 30 22:04:57.036181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.036153 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-openshift-ca" pod "9edbffec-25c8-483f-aa6a-259737a28b10" (UID: "9edbffec-25c8-483f-aa6a-259737a28b10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:57.058800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.058771 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" event={"ID":"294ec9c2-60c0-4b7d-a6af-2cfdc7478467","Type":"ContainerStarted","Data":"daeddef94804ea8918c22d08836308c6d5682672a8b370c9404f43ea98feb563"} Oct 30 22:04:57.061733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.061673 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:04:57.105512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.087182 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" event={"ID":"9edbffec-25c8-483f-aa6a-259737a28b10","Type":"ContainerDied","Data":"469c435ce67b68fca35e22f32c29ab4b97bf2bfcc3982079caef58735c5a7d6c"} Oct 30 22:04:57.105512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.087241 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469c435ce67b68fca35e22f32c29ab4b97bf2bfcc3982079caef58735c5a7d6c" Oct 30 22:04:57.105512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.087307 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p" Oct 30 22:04:57.223510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.216194 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:57.320071 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.320029 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6648aab-b4lzp\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:57.405656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.382207 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:57.405656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.382570 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:57.415500 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.409611 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:04:57.461371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.461329 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:57.483774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.472690 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-xks-ca" pod "9edbffec-25c8-483f-aa6a-259737a28b10" (UID: "9edbffec-25c8-483f-aa6a-259737a28b10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:04:57.518858 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.515961 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edbffec-25c8-483f-aa6a-259737a28b10-kube-api-access-gslbk" pod "9edbffec-25c8-483f-aa6a-259737a28b10" (UID: "9edbffec-25c8-483f-aa6a-259737a28b10"). InnerVolumeSpecName "kube-api-access-gslbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:57.550408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.550340 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9edbffec-25c8-483f-aa6a-259737a28b10-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:57.550408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.550373 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gslbk\" (UniqueName: \"kubernetes.io/projected/9edbffec-25c8-483f-aa6a-259737a28b10-kube-api-access-gslbk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:04:57.568409 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.568244 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:57.629536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.621297 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:04:57.629536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.621350 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 22:04:57.710020 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.709980 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:57.741738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.741647 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:04:57.741894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.741745 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:04:57.741894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.741808 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:04:57.742073 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.741906 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:04:57.860736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.860705 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:04:57.881516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.879383 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:57.906170 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.893400 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7305-7d99e9"] Oct 30 22:04:57.906170 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.905284 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:57.965522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.941328 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9edbffec-25c8-483f-aa6a-259737a28b10" containerName="revoke-keys" Oct 30 22:04:57.965797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.965770 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edbffec-25c8-483f-aa6a-259737a28b10" containerName="revoke-keys" Oct 30 22:04:57.997915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:57.997821 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edbffec-25c8-483f-aa6a-259737a28b10" containerName="revoke-keys" Oct 30 22:04:58.009026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.008982 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:58.027622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.008511 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:58.027622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.023611 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:58.027622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.027267 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:58.065452 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.065421 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:04:58.081616 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.081573 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 22:04:58.252758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.252665 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" event={"ID":"294ec9c2-60c0-4b7d-a6af-2cfdc7478467","Type":"ContainerStarted","Data":"7d71f3a6135ac10e495fe5a2bcef9e2d9517407838f5390ded4b3bbaf6e92a32"} Oct 30 22:04:58.253171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.253152 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:04:58.253364 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.253352 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:04:58.253638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.253035 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:04:58.411252 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.397684 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmjfd\"" Oct 30 22:04:58.440830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.439864 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:04:58.576534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.528599 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tm5w\" (UniqueName: \"kubernetes.io/projected/b72d0792-875e-4625-8162-c719b01bc0e3-kube-api-access-9tm5w\") pod \"test-curl-7305-7d99e9\" (UID: \"b72d0792-875e-4625-8162-c719b01bc0e3\") " pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:04:58.631573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.628138 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7305-7d99e9"] Oct 30 22:04:58.698467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.692783 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:04:58.775186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.770635 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:04:58.780594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.779623 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:58.780594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.779672 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:58.837371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.837337 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tm5w\" (UniqueName: \"kubernetes.io/projected/b72d0792-875e-4625-8162-c719b01bc0e3-kube-api-access-9tm5w\") pod \"test-curl-7305-7d99e9\" (UID: \"b72d0792-875e-4625-8162-c719b01bc0e3\") " pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:04:58.914667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.914640 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:58.917528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.917508 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:04:58.956553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.928817 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:04:58.956553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:58.936529 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tm5w\" (UniqueName: \"kubernetes.io/projected/b72d0792-875e-4625-8162-c719b01bc0e3-kube-api-access-9tm5w\") pod \"test-curl-7305-7d99e9\" (UID: \"b72d0792-875e-4625-8162-c719b01bc0e3\") " pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:04:59.081831 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.081801 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:04:59.360934 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.357100 4933 generic.go:356] "Generic (PLEG): container finished" podID="294ec9c2-60c0-4b7d-a6af-2cfdc7478467" containerID="7d71f3a6135ac10e495fe5a2bcef9e2d9517407838f5390ded4b3bbaf6e92a32" exitCode=0 Oct 30 22:04:59.403592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.363052 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" event={"ID":"294ec9c2-60c0-4b7d-a6af-2cfdc7478467","Type":"ContainerDied","Data":"7d71f3a6135ac10e495fe5a2bcef9e2d9517407838f5390ded4b3bbaf6e92a32"} Oct 30 22:04:59.403592 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:04:59.388675 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:04:59.403592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.395605 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" event={"ID":"79ded627-4bad-4d13-96a2-fef1fb1b7452","Type":"ContainerStarted","Data":"31aa7003b063fd9ac9dfa3afa8dc1af898843df86734372dce6a5604c46eab2d"} Oct 30 22:04:59.419023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.418422 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:59.451311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.448808 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp"] Oct 30 22:04:59.451311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.448930 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:04:59.576436 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.576230 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" podStartSLOduration=10.576210944 podStartE2EDuration="10.576210944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:59.409939382 +0000 UTC m=+1348.091089279" watchObservedRunningTime="2026-10-30 22:04:59.576210944 +0000 UTC m=+1348.257360841" Oct 30 22:04:59.615189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.611671 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:59.615189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.611740 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:59.682527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.681101 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:04:59.685521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:04:59.685292 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:05:00.013606 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:00.013576 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb72d0792_875e_4625_8162_c719b01bc0e3.slice/crio-0b8228ad63ce2fcb683f132ae3a1dd4e2c5bac4963b85af8ab91a322bdf5ac8e WatchSource:0}: Error finding container 0b8228ad63ce2fcb683f132ae3a1dd4e2c5bac4963b85af8ab91a322bdf5ac8e: Status 404 returned error can't find the container with id 0b8228ad63ce2fcb683f132ae3a1dd4e2c5bac4963b85af8ab91a322bdf5ac8e Oct 30 22:05:00.016103 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:00.016078 4933 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:05:00.017756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:00.017706 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7305-7d99e9"] Oct 30 22:05:00.676561 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:00.672638 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:00.676561 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:00.672706 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:00.886816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:00.886774 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7305-7d99e9" event={"ID":"b72d0792-875e-4625-8162-c719b01bc0e3","Type":"ContainerStarted","Data":"0b8228ad63ce2fcb683f132ae3a1dd4e2c5bac4963b85af8ab91a322bdf5ac8e"} Oct 30 22:05:00.954246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:00.912374 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" event={"ID":"79ded627-4bad-4d13-96a2-fef1fb1b7452","Type":"ContainerStarted","Data":"cae0c3bb65d7f9f19bf8391d81e41e7ab4231364d38ae01d940cbf5937a8b53b"} Oct 30 22:05:01.103050 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:01.103010 4933 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ded627_4bad_4d13_96a2_fef1fb1b7452.slice/crio-cae0c3bb65d7f9f19bf8391d81e41e7ab4231364d38ae01d940cbf5937a8b53b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ded627_4bad_4d13_96a2_fef1fb1b7452.slice/crio-conmon-cae0c3bb65d7f9f19bf8391d81e41e7ab4231364d38ae01d940cbf5937a8b53b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:05:01.282515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.282122 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:05:01.285377 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.285268 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:05:01.408481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.408456 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:05:01.501390 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.501312 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-kube-api-access-nqmwf\" (UniqueName: \"kubernetes.io/projected/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-kube-api-access-nqmwf\") pod \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " Oct 30 22:05:01.501577 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.501426 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-xks-ca\") pod \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " Oct 30 22:05:01.501663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.501589 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-openshift-ca\") pod \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\" (UID: \"294ec9c2-60c0-4b7d-a6af-2cfdc7478467\") " Oct 30 22:05:01.501904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.501881 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-openshift-ca" pod "294ec9c2-60c0-4b7d-a6af-2cfdc7478467" (UID: "294ec9c2-60c0-4b7d-a6af-2cfdc7478467"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:01.502207 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.502189 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:01.503725 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.503692 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-xks-ca" pod "294ec9c2-60c0-4b7d-a6af-2cfdc7478467" (UID: "294ec9c2-60c0-4b7d-a6af-2cfdc7478467"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:01.503836 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.503817 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-kube-api-access-nqmwf" pod "294ec9c2-60c0-4b7d-a6af-2cfdc7478467" (UID: "294ec9c2-60c0-4b7d-a6af-2cfdc7478467"). InnerVolumeSpecName "kube-api-access-nqmwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:01.605079 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.605017 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:01.605079 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.605063 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqmwf\" (UniqueName: \"kubernetes.io/projected/294ec9c2-60c0-4b7d-a6af-2cfdc7478467-kube-api-access-nqmwf\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:01.922986 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.922948 4933 generic.go:356] "Generic (PLEG): container finished" podID="79ded627-4bad-4d13-96a2-fef1fb1b7452" containerID="cae0c3bb65d7f9f19bf8391d81e41e7ab4231364d38ae01d940cbf5937a8b53b" exitCode=0 Oct 30 22:05:01.923436 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.923027 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" event={"ID":"79ded627-4bad-4d13-96a2-fef1fb1b7452","Type":"ContainerDied","Data":"cae0c3bb65d7f9f19bf8391d81e41e7ab4231364d38ae01d940cbf5937a8b53b"} Oct 30 22:05:01.931248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.924895 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" Oct 30 22:05:01.931248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.924957 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz" event={"ID":"294ec9c2-60c0-4b7d-a6af-2cfdc7478467","Type":"ContainerDied","Data":"daeddef94804ea8918c22d08836308c6d5682672a8b370c9404f43ea98feb563"} Oct 30 22:05:01.931248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:01.925007 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daeddef94804ea8918c22d08836308c6d5682672a8b370c9404f43ea98feb563" Oct 30 22:05:02.054168 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.054138 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz"] Oct 30 22:05:02.055036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.055009 4933 cpu_manager.go:414] "containerMap: removing container" podUID="294ec9c2-60c0-4b7d-a6af-2cfdc7478467" containerName="revoke-keys" Oct 30 22:05:02.055211 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.055187 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294ec9c2-60c0-4b7d-a6af-2cfdc7478467" containerName="revoke-keys" Oct 30 22:05:02.055622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.055427 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="294ec9c2-60c0-4b7d-a6af-2cfdc7478467" containerName="revoke-keys" Oct 30 22:05:02.060158 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.060134 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.114871 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.114847 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz"] Oct 30 22:05:02.119875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.119789 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.120045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.119908 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.120115 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.120094 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66x2j\" (UniqueName: \"kubernetes.io/projected/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-kube-api-access-66x2j\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.155372 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.155346 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:05:02.220762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.220695 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-xks-ca\") pod \"79ded627-4bad-4d13-96a2-fef1fb1b7452\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " Oct 30 22:05:02.220904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.220771 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ded627-4bad-4d13-96a2-fef1fb1b7452-kube-api-access-qwjzf\" (UniqueName: \"kubernetes.io/projected/79ded627-4bad-4d13-96a2-fef1fb1b7452-kube-api-access-qwjzf\") pod \"79ded627-4bad-4d13-96a2-fef1fb1b7452\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " Oct 30 22:05:02.221029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.221012 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-openshift-ca\") pod \"79ded627-4bad-4d13-96a2-fef1fb1b7452\" (UID: \"79ded627-4bad-4d13-96a2-fef1fb1b7452\") " Oct 30 22:05:02.221240 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.221223 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.221427 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.221412 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66x2j\" (UniqueName: \"kubernetes.io/projected/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-kube-api-access-66x2j\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.221648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.221612 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.221648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.221634 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-openshift-ca" pod "79ded627-4bad-4d13-96a2-fef1fb1b7452" (UID: "79ded627-4bad-4d13-96a2-fef1fb1b7452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:02.222443 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.222418 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.223337 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.223302 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-xks-ca" pod "79ded627-4bad-4d13-96a2-fef1fb1b7452" (UID: "79ded627-4bad-4d13-96a2-fef1fb1b7452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:02.223934 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.223886 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ded627-4bad-4d13-96a2-fef1fb1b7452-kube-api-access-qwjzf" pod "79ded627-4bad-4d13-96a2-fef1fb1b7452" (UID: "79ded627-4bad-4d13-96a2-fef1fb1b7452"). InnerVolumeSpecName "kube-api-access-qwjzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:02.224843 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.224818 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.280278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.280253 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66x2j\" (UniqueName: \"kubernetes.io/projected/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-kube-api-access-66x2j\") pod \"maas-api-revoke-sub-090d71e8-lv9dz\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.323831 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.323798 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:02.324019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.323837 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ded627-4bad-4d13-96a2-fef1fb1b7452-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:02.324019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.323852 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwjzf\" (UniqueName: \"kubernetes.io/projected/79ded627-4bad-4d13-96a2-fef1fb1b7452-kube-api-access-qwjzf\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:02.345538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.341048 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n"] Oct 30 22:05:02.345538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.341754 4933 cpu_manager.go:414] "containerMap: removing container" podUID="79ded627-4bad-4d13-96a2-fef1fb1b7452" containerName="revoke-keys" Oct 30 22:05:02.345538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.341768 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ded627-4bad-4d13-96a2-fef1fb1b7452" containerName="revoke-keys" Oct 30 22:05:02.345538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.341925 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ded627-4bad-4d13-96a2-fef1fb1b7452" containerName="revoke-keys" Oct 30 22:05:02.345850 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.345776 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.376624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.376600 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:02.399712 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.399683 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n"] Oct 30 22:05:02.443460 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.443425 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zchd\" (UniqueName: \"kubernetes.io/projected/23f60c16-dd5b-40e8-9c68-fa611500376f-kube-api-access-5zchd\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.443649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.443480 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.443779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.443762 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.545467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.545231 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zchd\" (UniqueName: \"kubernetes.io/projected/23f60c16-dd5b-40e8-9c68-fa611500376f-kube-api-access-5zchd\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.545467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.545287 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.546193 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.546065 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.547182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.546888 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.550050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.550012 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.611148 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.611122 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zchd\" (UniqueName: \"kubernetes.io/projected/23f60c16-dd5b-40e8-9c68-fa611500376f-kube-api-access-5zchd\") pod \"maas-api-revoke-sub-d12c7586-hxh8n\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.666883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.666666 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:02.811933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.811884 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz"] Oct 30 22:05:02.821088 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:02.821053 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e06ae8_93a9_43f6_a9cd_8de09b85fb2f.slice/crio-57665c6b383406754b5b6bf774d0e1685591f4b4ba273808264d93a99162b621 WatchSource:0}: Error finding container 57665c6b383406754b5b6bf774d0e1685591f4b4ba273808264d93a99162b621: Status 404 returned error can't find the container with id 57665c6b383406754b5b6bf774d0e1685591f4b4ba273808264d93a99162b621 Oct 30 22:05:02.938756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.938731 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" Oct 30 22:05:02.939188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.938736 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp" event={"ID":"79ded627-4bad-4d13-96a2-fef1fb1b7452","Type":"ContainerDied","Data":"31aa7003b063fd9ac9dfa3afa8dc1af898843df86734372dce6a5604c46eab2d"} Oct 30 22:05:02.939188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.938859 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31aa7003b063fd9ac9dfa3afa8dc1af898843df86734372dce6a5604c46eab2d" Oct 30 22:05:02.940696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:02.940666 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" event={"ID":"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f","Type":"ContainerStarted","Data":"57665c6b383406754b5b6bf774d0e1685591f4b4ba273808264d93a99162b621"} Oct 30 22:05:03.071189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:03.071161 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n"] Oct 30 22:05:03.953126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:03.953092 4933 generic.go:356] "Generic (PLEG): container finished" podID="79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" containerID="c9d6665d56d3f4e29170d8ef3ef69f7dba3c9709d221ac6008d37c83c26d6b65" exitCode=0 Oct 30 22:05:03.953587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:03.953171 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" event={"ID":"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f","Type":"ContainerDied","Data":"c9d6665d56d3f4e29170d8ef3ef69f7dba3c9709d221ac6008d37c83c26d6b65"} Oct 30 22:05:03.955167 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:03.955140 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7305-7d99e9" event={"ID":"b72d0792-875e-4625-8162-c719b01bc0e3","Type":"ContainerStarted","Data":"bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5"} Oct 30 22:05:03.957067 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:03.957039 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" event={"ID":"23f60c16-dd5b-40e8-9c68-fa611500376f","Type":"ContainerStarted","Data":"b0890bd965f27f5ec9976aa7785d51e853c78e157d12f98728482239c1b34c32"} Oct 30 22:05:03.957161 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:03.957088 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" event={"ID":"23f60c16-dd5b-40e8-9c68-fa611500376f","Type":"ContainerStarted","Data":"7cebec05fa0e488b3f4cb8c9a38c6a76e89db232b3ed39368076563ad4bdc745"} Oct 30 22:05:04.191269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:04.191210 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" podStartSLOduration=2.191191411 podStartE2EDuration="2.191191411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:04.183950337 +0000 UTC m=+1352.865100235" watchObservedRunningTime="2026-10-30 22:05:04.191191411 +0000 UTC m=+1352.872341307" Oct 30 22:05:04.298330 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:04.298281 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7305-7d99e9" podStartSLOduration=3.631556265 podStartE2EDuration="7.298267194s" totalImagesPullingTime="3.666710929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:04.297638026 +0000 UTC m=+1352.978787921" watchObservedRunningTime="2026-10-30 22:05:04.298267194 +0000 UTC m=+1352.979417112" Oct 30 22:05:04.966569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:04.966537 4933 generic.go:356] "Generic (PLEG): container finished" podID="23f60c16-dd5b-40e8-9c68-fa611500376f" containerID="b0890bd965f27f5ec9976aa7785d51e853c78e157d12f98728482239c1b34c32" exitCode=0 Oct 30 22:05:04.966962 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:04.966628 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" event={"ID":"23f60c16-dd5b-40e8-9c68-fa611500376f","Type":"ContainerDied","Data":"b0890bd965f27f5ec9976aa7785d51e853c78e157d12f98728482239c1b34c32"} Oct 30 22:05:05.178848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.178827 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:05.291912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.291889 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-openshift-ca\") pod \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " Oct 30 22:05:05.292138 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.292122 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-kube-api-access-66x2j\" (UniqueName: \"kubernetes.io/projected/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-kube-api-access-66x2j\") pod \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " Oct 30 22:05:05.292205 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.292151 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-openshift-ca" pod "79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" (UID: "79e06ae8-93a9-43f6-a9cd-8de09b85fb2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:05.292283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.292268 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-xks-ca\") pod \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\" (UID: \"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f\") " Oct 30 22:05:05.292778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.292759 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.294447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.294423 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-xks-ca" pod "79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" (UID: "79e06ae8-93a9-43f6-a9cd-8de09b85fb2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:05.294575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.294455 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-kube-api-access-66x2j" pod "79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" (UID: "79e06ae8-93a9-43f6-a9cd-8de09b85fb2f"). InnerVolumeSpecName "kube-api-access-66x2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:05.393747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.393719 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66x2j\" (UniqueName: \"kubernetes.io/projected/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-kube-api-access-66x2j\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.393747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.393746 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:05.719328 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.719231 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7305-7d99e9"] Oct 30 22:05:05.724707 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.724669 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7305-7d99e9"] Oct 30 22:05:05.883689 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.883660 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7305-624e02"] Oct 30 22:05:05.884346 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.884326 4933 cpu_manager.go:414] "containerMap: removing container" podUID="79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" containerName="revoke-keys" Oct 30 22:05:05.884477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.884348 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" containerName="revoke-keys" Oct 30 22:05:05.884477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.884408 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" containerName="test-curl-7305-7d99e9" Oct 30 22:05:05.884477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.884416 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" containerName="test-curl-7305-7d99e9" Oct 30 22:05:05.884654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.884610 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" containerName="test-curl-7305-7d99e9" Oct 30 22:05:05.884654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.884635 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" containerName="revoke-keys" Oct 30 22:05:05.899963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.899935 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:05.936560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.936532 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7305-624e02"] Oct 30 22:05:05.977653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.977581 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" event={"ID":"79e06ae8-93a9-43f6-a9cd-8de09b85fb2f","Type":"ContainerDied","Data":"57665c6b383406754b5b6bf774d0e1685591f4b4ba273808264d93a99162b621"} Oct 30 22:05:05.977653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.977633 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57665c6b383406754b5b6bf774d0e1685591f4b4ba273808264d93a99162b621" Oct 30 22:05:05.977653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.977590 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz" Oct 30 22:05:05.978139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:05.978111 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7305-7d99e9" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" containerName="test-curl-7305-7d99e9" containerID="cri-o://bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5" gracePeriod=2 Oct 30 22:05:06.001552 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.001477 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sz5c\" (UniqueName: \"kubernetes.io/projected/9692f3bc-8e05-4a4e-8022-b39a84bdc57f-kube-api-access-9sz5c\") pod \"test-curl-7305-624e02\" (UID: \"9692f3bc-8e05-4a4e-8022-b39a84bdc57f\") " pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:06.103663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.103630 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sz5c\" (UniqueName: \"kubernetes.io/projected/9692f3bc-8e05-4a4e-8022-b39a84bdc57f-kube-api-access-9sz5c\") pod \"test-curl-7305-624e02\" (UID: \"9692f3bc-8e05-4a4e-8022-b39a84bdc57f\") " pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:06.121701 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:06.121675 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-7d99e9\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:06.144387 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.144365 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:06.162041 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.162011 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sz5c\" (UniqueName: \"kubernetes.io/projected/9692f3bc-8e05-4a4e-8022-b39a84bdc57f-kube-api-access-9sz5c\") pod \"test-curl-7305-624e02\" (UID: \"9692f3bc-8e05-4a4e-8022-b39a84bdc57f\") " pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:06.204646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.204624 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-openshift-ca\") pod \"23f60c16-dd5b-40e8-9c68-fa611500376f\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " Oct 30 22:05:06.204812 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.204800 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f60c16-dd5b-40e8-9c68-fa611500376f-kube-api-access-5zchd\" (UniqueName: \"kubernetes.io/projected/23f60c16-dd5b-40e8-9c68-fa611500376f-kube-api-access-5zchd\") pod \"23f60c16-dd5b-40e8-9c68-fa611500376f\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " Oct 30 22:05:06.204872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.204859 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-xks-ca\") pod \"23f60c16-dd5b-40e8-9c68-fa611500376f\" (UID: \"23f60c16-dd5b-40e8-9c68-fa611500376f\") " Oct 30 22:05:06.204955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.204931 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-openshift-ca" pod "23f60c16-dd5b-40e8-9c68-fa611500376f" (UID: "23f60c16-dd5b-40e8-9c68-fa611500376f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:06.205363 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.205346 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:06.206941 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.206913 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f60c16-dd5b-40e8-9c68-fa611500376f-kube-api-access-5zchd" pod "23f60c16-dd5b-40e8-9c68-fa611500376f" (UID: "23f60c16-dd5b-40e8-9c68-fa611500376f"). InnerVolumeSpecName "kube-api-access-5zchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:06.207102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.207077 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-xks-ca" pod "23f60c16-dd5b-40e8-9c68-fa611500376f" (UID: "23f60c16-dd5b-40e8-9c68-fa611500376f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:06.213241 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.213226 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:06.227733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.227692 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn"] Oct 30 22:05:06.228344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.228329 4933 cpu_manager.go:414] "containerMap: removing container" podUID="23f60c16-dd5b-40e8-9c68-fa611500376f" containerName="revoke-keys" Oct 30 22:05:06.228391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.228347 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f60c16-dd5b-40e8-9c68-fa611500376f" containerName="revoke-keys" Oct 30 22:05:06.228457 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.228448 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f60c16-dd5b-40e8-9c68-fa611500376f" containerName="revoke-keys" Oct 30 22:05:06.233230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.233214 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.296093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.294722 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn"] Oct 30 22:05:06.306653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.306625 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.306970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.306946 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.307092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.307074 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5fs\" (UniqueName: \"kubernetes.io/projected/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-kube-api-access-4m5fs\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.307234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.307221 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zchd\" (UniqueName: \"kubernetes.io/projected/23f60c16-dd5b-40e8-9c68-fa611500376f-kube-api-access-5zchd\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:06.307332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.307242 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f60c16-dd5b-40e8-9c68-fa611500376f-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:06.315600 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:06.315573 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-7d99e9\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:06.408028 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.407995 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.408174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.408082 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5fs\" (UniqueName: \"kubernetes.io/projected/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-kube-api-access-4m5fs\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.408251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.408207 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.408872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.408852 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.410414 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.410398 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.467377 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.467356 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5fs\" (UniqueName: \"kubernetes.io/projected/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-kube-api-access-4m5fs\") pod \"maas-api-revoke-sub-fc8b8bfe-9z7zn\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.483635 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:06.483577 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-7d99e9\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:06.558031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.557990 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:06.598914 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:06.598881 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9692f3bc_8e05_4a4e_8022_b39a84bdc57f.slice/crio-7d65828f4eddf3a25e5c27bc28352dea48a7f3fcab6d921a72511538e8ae9b3d WatchSource:0}: Error finding container 7d65828f4eddf3a25e5c27bc28352dea48a7f3fcab6d921a72511538e8ae9b3d: Status 404 returned error can't find the container with id 7d65828f4eddf3a25e5c27bc28352dea48a7f3fcab6d921a72511538e8ae9b3d Oct 30 22:05:06.601247 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.601227 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7305-624e02"] Oct 30 22:05:06.939480 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:06.939440 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d483ee_1f1d_4a19_bbdf_7ef619280ec4.slice/crio-148d18bbbc6b679dd0571051b3a50b49c9d69bfa84f54fc127e8c0110d744d3d WatchSource:0}: Error finding container 148d18bbbc6b679dd0571051b3a50b49c9d69bfa84f54fc127e8c0110d744d3d: Status 404 returned error can't find the container with id 148d18bbbc6b679dd0571051b3a50b49c9d69bfa84f54fc127e8c0110d744d3d Oct 30 22:05:06.943916 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.943889 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn"] Oct 30 22:05:06.990463 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.990438 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7305-624e02" event={"ID":"9692f3bc-8e05-4a4e-8022-b39a84bdc57f","Type":"ContainerStarted","Data":"f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b"} Oct 30 22:05:06.990912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.990483 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7305-624e02" event={"ID":"9692f3bc-8e05-4a4e-8022-b39a84bdc57f","Type":"ContainerStarted","Data":"7d65828f4eddf3a25e5c27bc28352dea48a7f3fcab6d921a72511538e8ae9b3d"} Oct 30 22:05:06.992000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.991975 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" event={"ID":"04d483ee-1f1d-4a19-bbdf-7ef619280ec4","Type":"ContainerStarted","Data":"148d18bbbc6b679dd0571051b3a50b49c9d69bfa84f54fc127e8c0110d744d3d"} Oct 30 22:05:06.993611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.993577 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" event={"ID":"23f60c16-dd5b-40e8-9c68-fa611500376f","Type":"ContainerDied","Data":"7cebec05fa0e488b3f4cb8c9a38c6a76e89db232b3ed39368076563ad4bdc745"} Oct 30 22:05:06.993611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.993594 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n" Oct 30 22:05:06.993770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:06.993633 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cebec05fa0e488b3f4cb8c9a38c6a76e89db232b3ed39368076563ad4bdc745" Oct 30 22:05:07.113350 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:07.113293 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7305-624e02" podStartSLOduration=2.113276297 podStartE2EDuration="2.113276297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:07.105609706 +0000 UTC m=+1355.786759599" watchObservedRunningTime="2026-10-30 22:05:07.113276297 +0000 UTC m=+1355.794426192" Oct 30 22:05:07.157356 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:07.157319 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-7d99e9\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:07.267248 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:07.267218 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-7d99e9\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:08.014184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.014143 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" event={"ID":"04d483ee-1f1d-4a19-bbdf-7ef619280ec4","Type":"ContainerStarted","Data":"b33759cd8047e30d3b7e196728804a23e820b398d41a4d92cf0a2ddfebaa2058"} Oct 30 22:05:08.137626 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.137570 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" podStartSLOduration=2.137551736 podStartE2EDuration="2.137551736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:08.131613386 +0000 UTC m=+1356.812763281" watchObservedRunningTime="2026-10-30 22:05:08.137551736 +0000 UTC m=+1356.818701630" Oct 30 22:05:08.254753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.254726 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:08.336995 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.336965 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72d0792-875e-4625-8162-c719b01bc0e3-kube-api-access-9tm5w\" (UniqueName: \"kubernetes.io/projected/b72d0792-875e-4625-8162-c719b01bc0e3-kube-api-access-9tm5w\") pod \"b72d0792-875e-4625-8162-c719b01bc0e3\" (UID: \"b72d0792-875e-4625-8162-c719b01bc0e3\") " Oct 30 22:05:08.339572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.339543 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72d0792-875e-4625-8162-c719b01bc0e3-kube-api-access-9tm5w" pod "b72d0792-875e-4625-8162-c719b01bc0e3" (UID: "b72d0792-875e-4625-8162-c719b01bc0e3"). InnerVolumeSpecName "kube-api-access-9tm5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:08.438813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.438780 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tm5w\" (UniqueName: \"kubernetes.io/projected/b72d0792-875e-4625-8162-c719b01bc0e3-kube-api-access-9tm5w\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:08.687504 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.687466 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7305-624e02"] Oct 30 22:05:08.699417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:08.699382 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7305-624e02"] Oct 30 22:05:09.023427 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.023331 4933 generic.go:356] "Generic (PLEG): container finished" podID="b72d0792-875e-4625-8162-c719b01bc0e3" containerID="bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5" exitCode=137 Oct 30 22:05:09.023427 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.023377 4933 scope.go:122] "RemoveContainer" containerID="bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5" Oct 30 22:05:09.023427 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.023384 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:09.025608 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.025584 4933 generic.go:356] "Generic (PLEG): container finished" podID="04d483ee-1f1d-4a19-bbdf-7ef619280ec4" containerID="b33759cd8047e30d3b7e196728804a23e820b398d41a4d92cf0a2ddfebaa2058" exitCode=0 Oct 30 22:05:09.025738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.025673 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" event={"ID":"04d483ee-1f1d-4a19-bbdf-7ef619280ec4","Type":"ContainerDied","Data":"b33759cd8047e30d3b7e196728804a23e820b398d41a4d92cf0a2ddfebaa2058"} Oct 30 22:05:09.025814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.025798 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7305-624e02" podUID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" containerName="test-curl-7305-624e02" containerID="cri-o://f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b" gracePeriod=2 Oct 30 22:05:09.035782 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.035760 4933 scope.go:122] "RemoveContainer" containerID="bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5" Oct 30 22:05:09.036077 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:09.036057 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5\": container with ID starting with bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5 not found: ID does not exist" containerID="bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5" Oct 30 22:05:09.036174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.036086 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5"} err="failed to get container status \"bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5\": rpc error: code = NotFound desc = could not find container \"bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5\": container with ID starting with bef4bf6d7f6258221647fc0a300c3c218927e4ba7d52012eec85aa9d997155c5 not found: ID does not exist" Oct 30 22:05:09.077814 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:09.077785 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-624e02\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:09.247348 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:09.247310 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-7d99e9\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" pod="opendatahub/test-curl-7305-7d99e9" Oct 30 22:05:09.299235 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:09.299204 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7305-624e02\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:09.468145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:09.468115 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72d0792-875e-4625-8162-c719b01bc0e3" path="/var/lib/kubelet/pods/b72d0792-875e-4625-8162-c719b01bc0e3/volumes" Oct 30 22:05:10.473528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.472384 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:10.615895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.615852 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-xks-ca\") pod \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " Oct 30 22:05:10.616080 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.615928 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-kube-api-access-4m5fs\" (UniqueName: \"kubernetes.io/projected/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-kube-api-access-4m5fs\") pod \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " Oct 30 22:05:10.616080 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.615971 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-openshift-ca\") pod \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\" (UID: \"04d483ee-1f1d-4a19-bbdf-7ef619280ec4\") " Oct 30 22:05:10.617586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.617554 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-openshift-ca" pod "04d483ee-1f1d-4a19-bbdf-7ef619280ec4" (UID: "04d483ee-1f1d-4a19-bbdf-7ef619280ec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:10.652970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.652934 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-kube-api-access-4m5fs" pod "04d483ee-1f1d-4a19-bbdf-7ef619280ec4" (UID: "04d483ee-1f1d-4a19-bbdf-7ef619280ec4"). InnerVolumeSpecName "kube-api-access-4m5fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:10.675475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.675076 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-xks-ca" pod "04d483ee-1f1d-4a19-bbdf-7ef619280ec4" (UID: "04d483ee-1f1d-4a19-bbdf-7ef619280ec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:10.719790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.719753 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:10.719790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.719792 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m5fs\" (UniqueName: \"kubernetes.io/projected/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-kube-api-access-4m5fs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:10.720050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:10.719809 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04d483ee-1f1d-4a19-bbdf-7ef619280ec4-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:11.052250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.052228 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" Oct 30 22:05:11.052250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.052236 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn" event={"ID":"04d483ee-1f1d-4a19-bbdf-7ef619280ec4","Type":"ContainerDied","Data":"148d18bbbc6b679dd0571051b3a50b49c9d69bfa84f54fc127e8c0110d744d3d"} Oct 30 22:05:11.052417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.052304 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="148d18bbbc6b679dd0571051b3a50b49c9d69bfa84f54fc127e8c0110d744d3d" Oct 30 22:05:11.274029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.274001 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:11.339127 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.339044 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9692f3bc-8e05-4a4e-8022-b39a84bdc57f-kube-api-access-9sz5c\" (UniqueName: \"kubernetes.io/projected/9692f3bc-8e05-4a4e-8022-b39a84bdc57f-kube-api-access-9sz5c\") pod \"9692f3bc-8e05-4a4e-8022-b39a84bdc57f\" (UID: \"9692f3bc-8e05-4a4e-8022-b39a84bdc57f\") " Oct 30 22:05:11.341269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.341243 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9692f3bc-8e05-4a4e-8022-b39a84bdc57f-kube-api-access-9sz5c" pod "9692f3bc-8e05-4a4e-8022-b39a84bdc57f" (UID: "9692f3bc-8e05-4a4e-8022-b39a84bdc57f"). InnerVolumeSpecName "kube-api-access-9sz5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:11.441383 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.441351 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sz5c\" (UniqueName: \"kubernetes.io/projected/9692f3bc-8e05-4a4e-8022-b39a84bdc57f-kube-api-access-9sz5c\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:11.469896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:11.469866 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" path="/var/lib/kubelet/pods/9692f3bc-8e05-4a4e-8022-b39a84bdc57f/volumes" Oct 30 22:05:12.067258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.067222 4933 generic.go:356] "Generic (PLEG): container finished" podID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" containerID="f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b" exitCode=137 Oct 30 22:05:12.067788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.067288 4933 scope.go:122] "RemoveContainer" containerID="f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b" Oct 30 22:05:12.067788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.067285 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7305-624e02" Oct 30 22:05:12.081200 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.081179 4933 scope.go:122] "RemoveContainer" containerID="f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b" Oct 30 22:05:12.081471 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:12.081449 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b\": container with ID starting with f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b not found: ID does not exist" containerID="f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b" Oct 30 22:05:12.081588 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.081480 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b"} err="failed to get container status \"f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b\": rpc error: code = NotFound desc = could not find container \"f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b\": container with ID starting with f158cf7749131ac1db003e16872d073217967d298165bb40c4892a82e17eff8b not found: ID does not exist" Oct 30 22:05:12.853995 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.853959 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7"] Oct 30 22:05:12.854651 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.854636 4933 cpu_manager.go:414] "containerMap: removing container" podUID="04d483ee-1f1d-4a19-bbdf-7ef619280ec4" containerName="revoke-keys" Oct 30 22:05:12.854702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.854654 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d483ee-1f1d-4a19-bbdf-7ef619280ec4" containerName="revoke-keys" Oct 30 22:05:12.854702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.854678 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" containerName="test-curl-7305-624e02" Oct 30 22:05:12.854702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.854683 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" containerName="test-curl-7305-624e02" Oct 30 22:05:12.854797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.854779 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d483ee-1f1d-4a19-bbdf-7ef619280ec4" containerName="revoke-keys" Oct 30 22:05:12.854797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.854790 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9692f3bc-8e05-4a4e-8022-b39a84bdc57f" containerName="test-curl-7305-624e02" Oct 30 22:05:12.859852 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.859829 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:12.914227 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.914188 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:12.919064 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.919034 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7"] Oct 30 22:05:12.960170 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.960145 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:12.960394 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.960163 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:05:12.960482 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.960206 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:12.965569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.964544 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lgz\" (UniqueName: \"kubernetes.io/projected/25cac580-21ff-424a-a265-cbde8e89c5d9-kube-api-access-q5lgz\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:12.968413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.964794 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:12.968413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:12.966826 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.068266 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.068236 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.068690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.068371 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lgz\" (UniqueName: \"kubernetes.io/projected/25cac580-21ff-424a-a265-cbde8e89c5d9-kube-api-access-q5lgz\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.068690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.068480 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.069021 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.069001 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.071377 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.071246 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.127353 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.127294 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lgz\" (UniqueName: \"kubernetes.io/projected/25cac580-21ff-424a-a265-cbde8e89c5d9-kube-api-access-q5lgz\") pod \"maas-api-revoke-sub-4f5cc6b1-fxjn7\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.176463 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.176437 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:13.208129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.208022 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf"] Oct 30 22:05:13.213471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.213451 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.272420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.270975 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.272420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.271189 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.272420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.271342 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfghp\" (UniqueName: \"kubernetes.io/projected/4e9df465-8f8c-4b67-a58b-69f43046c734-kube-api-access-jfghp\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.272719 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.272673 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf"] Oct 30 22:05:13.372892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.372864 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.373037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.372948 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfghp\" (UniqueName: \"kubernetes.io/projected/4e9df465-8f8c-4b67-a58b-69f43046c734-kube-api-access-jfghp\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.373037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.372989 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.373620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.373602 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.375449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.375429 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.481207 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.481143 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfghp\" (UniqueName: \"kubernetes.io/projected/4e9df465-8f8c-4b67-a58b-69f43046c734-kube-api-access-jfghp\") pod \"maas-api-revoke-sub-f771a7e9-b58nf\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.528610 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.528584 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:13.569814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.569751 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7"] Oct 30 22:05:13.947425 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:13.941432 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9df465_8f8c_4b67_a58b_69f43046c734.slice/crio-4ab4175b32ea03f15ceae26df39dfe7ec2edce0ac9190ba2fcdf3d85a663e240 WatchSource:0}: Error finding container 4ab4175b32ea03f15ceae26df39dfe7ec2edce0ac9190ba2fcdf3d85a663e240: Status 404 returned error can't find the container with id 4ab4175b32ea03f15ceae26df39dfe7ec2edce0ac9190ba2fcdf3d85a663e240 Oct 30 22:05:13.947425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:13.943268 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf"] Oct 30 22:05:14.094596 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:14.094543 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" event={"ID":"4e9df465-8f8c-4b67-a58b-69f43046c734","Type":"ContainerStarted","Data":"4ab4175b32ea03f15ceae26df39dfe7ec2edce0ac9190ba2fcdf3d85a663e240"} Oct 30 22:05:14.096584 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:14.096540 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" event={"ID":"25cac580-21ff-424a-a265-cbde8e89c5d9","Type":"ContainerStarted","Data":"e8a436ab77d8d1de24f4b467e79c8a6f0a72f721ee74c99e73eb929d360e13b3"} Oct 30 22:05:14.096747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:14.096606 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" event={"ID":"25cac580-21ff-424a-a265-cbde8e89c5d9","Type":"ContainerStarted","Data":"e2b47ada8436f6d4549fb727c2b34841884bea4af825b412147acf8371e075a6"} Oct 30 22:05:14.217317 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:14.217268 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" podStartSLOduration=2.21725442 podStartE2EDuration="2.21725442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:14.215757167 +0000 UTC m=+1362.896907059" watchObservedRunningTime="2026-10-30 22:05:14.21725442 +0000 UTC m=+1362.898404313" Oct 30 22:05:15.109677 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:15.109643 4933 generic.go:356] "Generic (PLEG): container finished" podID="4e9df465-8f8c-4b67-a58b-69f43046c734" containerID="394b49615d2ea5bafc2d8d98f14af40cf41303601a461ab3ba9419fa965abc1a" exitCode=0 Oct 30 22:05:15.110197 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:15.109724 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" event={"ID":"4e9df465-8f8c-4b67-a58b-69f43046c734","Type":"ContainerDied","Data":"394b49615d2ea5bafc2d8d98f14af40cf41303601a461ab3ba9419fa965abc1a"} Oct 30 22:05:15.116905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:15.116875 4933 generic.go:356] "Generic (PLEG): container finished" podID="25cac580-21ff-424a-a265-cbde8e89c5d9" containerID="e8a436ab77d8d1de24f4b467e79c8a6f0a72f721ee74c99e73eb929d360e13b3" exitCode=0 Oct 30 22:05:15.117079 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:15.116959 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" event={"ID":"25cac580-21ff-424a-a265-cbde8e89c5d9","Type":"ContainerDied","Data":"e8a436ab77d8d1de24f4b467e79c8a6f0a72f721ee74c99e73eb929d360e13b3"} Oct 30 22:05:16.372581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.372556 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:16.376361 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.376334 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:16.429665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.429607 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cac580-21ff-424a-a265-cbde8e89c5d9-kube-api-access-q5lgz\" (UniqueName: \"kubernetes.io/projected/25cac580-21ff-424a-a265-cbde8e89c5d9-kube-api-access-q5lgz\") pod \"25cac580-21ff-424a-a265-cbde8e89c5d9\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " Oct 30 22:05:16.429665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.429654 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-openshift-ca\") pod \"25cac580-21ff-424a-a265-cbde8e89c5d9\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " Oct 30 22:05:16.429847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.429705 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-openshift-ca\") pod \"4e9df465-8f8c-4b67-a58b-69f43046c734\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " Oct 30 22:05:16.429847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.429729 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-xks-ca\") pod \"25cac580-21ff-424a-a265-cbde8e89c5d9\" (UID: \"25cac580-21ff-424a-a265-cbde8e89c5d9\") " Oct 30 22:05:16.429847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.429810 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9df465-8f8c-4b67-a58b-69f43046c734-kube-api-access-jfghp\" (UniqueName: \"kubernetes.io/projected/4e9df465-8f8c-4b67-a58b-69f43046c734-kube-api-access-jfghp\") pod \"4e9df465-8f8c-4b67-a58b-69f43046c734\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " Oct 30 22:05:16.430039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.430021 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-xks-ca\") pod \"4e9df465-8f8c-4b67-a58b-69f43046c734\" (UID: \"4e9df465-8f8c-4b67-a58b-69f43046c734\") " Oct 30 22:05:16.430116 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.430057 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-openshift-ca" pod "4e9df465-8f8c-4b67-a58b-69f43046c734" (UID: "4e9df465-8f8c-4b67-a58b-69f43046c734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:16.430116 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.430073 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-openshift-ca" pod "25cac580-21ff-424a-a265-cbde8e89c5d9" (UID: "25cac580-21ff-424a-a265-cbde8e89c5d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:16.430644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.430619 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:16.430644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.430645 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:16.432481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.432451 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-xks-ca" pod "25cac580-21ff-424a-a265-cbde8e89c5d9" (UID: "25cac580-21ff-424a-a265-cbde8e89c5d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:16.432481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.432467 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cac580-21ff-424a-a265-cbde8e89c5d9-kube-api-access-q5lgz" pod "25cac580-21ff-424a-a265-cbde8e89c5d9" (UID: "25cac580-21ff-424a-a265-cbde8e89c5d9"). InnerVolumeSpecName "kube-api-access-q5lgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:16.432773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.432746 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-xks-ca" pod "4e9df465-8f8c-4b67-a58b-69f43046c734" (UID: "4e9df465-8f8c-4b67-a58b-69f43046c734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:16.432933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.432914 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9df465-8f8c-4b67-a58b-69f43046c734-kube-api-access-jfghp" pod "4e9df465-8f8c-4b67-a58b-69f43046c734" (UID: "4e9df465-8f8c-4b67-a58b-69f43046c734"). InnerVolumeSpecName "kube-api-access-jfghp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:16.532143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.532107 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9df465-8f8c-4b67-a58b-69f43046c734-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:16.532292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.532147 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lgz\" (UniqueName: \"kubernetes.io/projected/25cac580-21ff-424a-a265-cbde8e89c5d9-kube-api-access-q5lgz\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:16.532292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.532163 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cac580-21ff-424a-a265-cbde8e89c5d9-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:16.532292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:16.532178 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfghp\" (UniqueName: \"kubernetes.io/projected/4e9df465-8f8c-4b67-a58b-69f43046c734-kube-api-access-jfghp\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:17.138720 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.138691 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" Oct 30 22:05:17.138720 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.138706 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7" event={"ID":"25cac580-21ff-424a-a265-cbde8e89c5d9","Type":"ContainerDied","Data":"e2b47ada8436f6d4549fb727c2b34841884bea4af825b412147acf8371e075a6"} Oct 30 22:05:17.138976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.138760 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b47ada8436f6d4549fb727c2b34841884bea4af825b412147acf8371e075a6" Oct 30 22:05:17.140349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.140323 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" event={"ID":"4e9df465-8f8c-4b67-a58b-69f43046c734","Type":"ContainerDied","Data":"4ab4175b32ea03f15ceae26df39dfe7ec2edce0ac9190ba2fcdf3d85a663e240"} Oct 30 22:05:17.140349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.140347 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf" Oct 30 22:05:17.140580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.140366 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ab4175b32ea03f15ceae26df39dfe7ec2edce0ac9190ba2fcdf3d85a663e240" Oct 30 22:05:17.205412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.205384 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf"] Oct 30 22:05:17.206372 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.206352 4933 cpu_manager.go:414] "containerMap: removing container" podUID="4e9df465-8f8c-4b67-a58b-69f43046c734" containerName="revoke-keys" Oct 30 22:05:17.206441 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.206377 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9df465-8f8c-4b67-a58b-69f43046c734" containerName="revoke-keys" Oct 30 22:05:17.206441 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.206411 4933 cpu_manager.go:414] "containerMap: removing container" podUID="25cac580-21ff-424a-a265-cbde8e89c5d9" containerName="revoke-keys" Oct 30 22:05:17.206441 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.206418 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cac580-21ff-424a-a265-cbde8e89c5d9" containerName="revoke-keys" Oct 30 22:05:17.206633 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.206620 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9df465-8f8c-4b67-a58b-69f43046c734" containerName="revoke-keys" Oct 30 22:05:17.206674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.206640 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cac580-21ff-424a-a265-cbde8e89c5d9" containerName="revoke-keys" Oct 30 22:05:17.212049 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.212028 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.284056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.284024 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:17.284262 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.284168 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:17.301811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.301774 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf"] Oct 30 22:05:17.318344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.318314 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:17.328294 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.328254 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:05:17.370692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.370661 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9fv\" (UniqueName: \"kubernetes.io/projected/c15b39c0-2619-4289-b628-dc2e1e25c341-kube-api-access-pc9fv\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.370925 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.370907 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.371010 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.370947 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.472712 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.472608 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9fv\" (UniqueName: \"kubernetes.io/projected/c15b39c0-2619-4289-b628-dc2e1e25c341-kube-api-access-pc9fv\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.473158 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.472837 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.473158 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.472875 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.474127 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.474101 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.476272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.476244 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.537293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.537222 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9fv\" (UniqueName: \"kubernetes.io/projected/c15b39c0-2619-4289-b628-dc2e1e25c341-kube-api-access-pc9fv\") pod \"maas-api-revoke-sub-0d60d8d1-5mktf\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.541660 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.541631 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:17.841713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.841682 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk"] Oct 30 22:05:17.853220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.852609 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:17.856740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.856714 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk"] Oct 30 22:05:17.910685 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.910656 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1d71f464\"" Oct 30 22:05:17.911329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:17.911310 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1d71f464-dockercfg-442wb\"" Oct 30 22:05:18.015403 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.015361 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6bf96d-06d8-4ea7-b59a-4e7659773409-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:18.015649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.015629 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd86\" (UniqueName: \"kubernetes.io/projected/0b6bf96d-06d8-4ea7-b59a-4e7659773409-kube-api-access-7bd86\") pod \"payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:18.125877 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.123981 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd86\" (UniqueName: \"kubernetes.io/projected/0b6bf96d-06d8-4ea7-b59a-4e7659773409-kube-api-access-7bd86\") pod \"payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:18.125877 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.124099 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6bf96d-06d8-4ea7-b59a-4e7659773409-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:18.125877 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.125616 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6bf96d-06d8-4ea7-b59a-4e7659773409-plugins-config-volume\") pod \"payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:18.130581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.130556 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf"] Oct 30 22:05:18.168807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.168760 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" event={"ID":"c15b39c0-2619-4289-b628-dc2e1e25c341","Type":"ContainerStarted","Data":"2c115b42a40947c2c82ff06f1963bcacd61fa4153b616b58be8b8d36ab844ec2"} Oct 30 22:05:18.199816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.199788 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd86\" (UniqueName: \"kubernetes.io/projected/0b6bf96d-06d8-4ea7-b59a-4e7659773409-kube-api-access-7bd86\") pod \"payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:18.496530 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.496505 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:18.879123 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:18.879094 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6bf96d_06d8_4ea7_b59a_4e7659773409.slice/crio-041d401372aca83e3b745c68dc11438dbde64dc99779c55681fb8da5f6f75653 WatchSource:0}: Error finding container 041d401372aca83e3b745c68dc11438dbde64dc99779c55681fb8da5f6f75653: Status 404 returned error can't find the container with id 041d401372aca83e3b745c68dc11438dbde64dc99779c55681fb8da5f6f75653 Oct 30 22:05:18.881657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:18.881630 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk"] Oct 30 22:05:19.179308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:19.179237 4933 generic.go:356] "Generic (PLEG): container finished" podID="c15b39c0-2619-4289-b628-dc2e1e25c341" containerID="87baebb315fe56a177e07803f7cb7b21885eda5f346d8fac438fea8d76016fa8" exitCode=0 Oct 30 22:05:19.179458 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:19.179328 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" event={"ID":"c15b39c0-2619-4289-b628-dc2e1e25c341","Type":"ContainerDied","Data":"87baebb315fe56a177e07803f7cb7b21885eda5f346d8fac438fea8d76016fa8"} Oct 30 22:05:19.183878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:19.183850 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" event={"ID":"0b6bf96d-06d8-4ea7-b59a-4e7659773409","Type":"ContainerStarted","Data":"313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8"} Oct 30 22:05:19.184551 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:19.183903 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" event={"ID":"0b6bf96d-06d8-4ea7-b59a-4e7659773409","Type":"ContainerStarted","Data":"041d401372aca83e3b745c68dc11438dbde64dc99779c55681fb8da5f6f75653"} Oct 30 22:05:19.184551 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:19.184016 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:19.483015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:19.482921 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" podStartSLOduration=2.482908784 podStartE2EDuration="2.482908784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:19.482324198 +0000 UTC m=+1368.163474092" watchObservedRunningTime="2026-10-30 22:05:19.482908784 +0000 UTC m=+1368.164058676" Oct 30 22:05:20.439196 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.438836 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:20.574329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.574296 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15b39c0-2619-4289-b628-dc2e1e25c341-kube-api-access-pc9fv\" (UniqueName: \"kubernetes.io/projected/c15b39c0-2619-4289-b628-dc2e1e25c341-kube-api-access-pc9fv\") pod \"c15b39c0-2619-4289-b628-dc2e1e25c341\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " Oct 30 22:05:20.574576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.574552 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-openshift-ca\") pod \"c15b39c0-2619-4289-b628-dc2e1e25c341\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " Oct 30 22:05:20.574743 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.574615 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-xks-ca\") pod \"c15b39c0-2619-4289-b628-dc2e1e25c341\" (UID: \"c15b39c0-2619-4289-b628-dc2e1e25c341\") " Oct 30 22:05:20.575420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.575391 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-openshift-ca" pod "c15b39c0-2619-4289-b628-dc2e1e25c341" (UID: "c15b39c0-2619-4289-b628-dc2e1e25c341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:20.577623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.577597 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-xks-ca" pod "c15b39c0-2619-4289-b628-dc2e1e25c341" (UID: "c15b39c0-2619-4289-b628-dc2e1e25c341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:20.579675 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.579647 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15b39c0-2619-4289-b628-dc2e1e25c341-kube-api-access-pc9fv" pod "c15b39c0-2619-4289-b628-dc2e1e25c341" (UID: "c15b39c0-2619-4289-b628-dc2e1e25c341"). InnerVolumeSpecName "kube-api-access-pc9fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:20.676962 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.676933 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc9fv\" (UniqueName: \"kubernetes.io/projected/c15b39c0-2619-4289-b628-dc2e1e25c341-kube-api-access-pc9fv\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:20.676962 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.676960 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:20.676962 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:20.676970 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c15b39c0-2619-4289-b628-dc2e1e25c341-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:21.218625 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:21.218600 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" Oct 30 22:05:21.218799 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:21.218632 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf" event={"ID":"c15b39c0-2619-4289-b628-dc2e1e25c341","Type":"ContainerDied","Data":"2c115b42a40947c2c82ff06f1963bcacd61fa4153b616b58be8b8d36ab844ec2"} Oct 30 22:05:21.218799 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:21.218675 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c115b42a40947c2c82ff06f1963bcacd61fa4153b616b58be8b8d36ab844ec2" Oct 30 22:05:23.034135 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.034096 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl"] Oct 30 22:05:23.036988 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.036959 4933 cpu_manager.go:414] "containerMap: removing container" podUID="c15b39c0-2619-4289-b628-dc2e1e25c341" containerName="revoke-keys" Oct 30 22:05:23.037253 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.037232 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15b39c0-2619-4289-b628-dc2e1e25c341" containerName="revoke-keys" Oct 30 22:05:23.037607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.037591 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15b39c0-2619-4289-b628-dc2e1e25c341" containerName="revoke-keys" Oct 30 22:05:23.052546 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.050675 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.097816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.097768 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl"] Oct 30 22:05:23.136781 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.136749 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-64086213\"" Oct 30 22:05:23.136974 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.136960 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-64086213-dockercfg-pzkhc\"" Oct 30 22:05:23.165910 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.165879 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m"] Oct 30 22:05:23.171836 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.171811 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.201539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.201515 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m"] Oct 30 22:05:23.218675 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.218649 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcm47\" (UniqueName: \"kubernetes.io/projected/ef410b18-912a-4d5f-a511-5115972a418b-kube-api-access-hcm47\") pod \"payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.218835 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.218707 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef410b18-912a-4d5f-a511-5115972a418b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.320561 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.320456 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrsk\" (UniqueName: \"kubernetes.io/projected/eaafe556-aa91-411a-bc1b-2baac76e8b40-kube-api-access-rhrsk\") pod \"payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.320716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.320579 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaafe556-aa91-411a-bc1b-2baac76e8b40-plugins-config-volume\") pod \"payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.320716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.320630 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcm47\" (UniqueName: \"kubernetes.io/projected/ef410b18-912a-4d5f-a511-5115972a418b-kube-api-access-hcm47\") pod \"payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.320849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.320764 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef410b18-912a-4d5f-a511-5115972a418b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.321411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.321392 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef410b18-912a-4d5f-a511-5115972a418b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.397373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.397338 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcm47\" (UniqueName: \"kubernetes.io/projected/ef410b18-912a-4d5f-a511-5115972a418b-kube-api-access-hcm47\") pod \"payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.424632 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.424594 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrsk\" (UniqueName: \"kubernetes.io/projected/eaafe556-aa91-411a-bc1b-2baac76e8b40-kube-api-access-rhrsk\") pod \"payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.424821 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.424673 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaafe556-aa91-411a-bc1b-2baac76e8b40-plugins-config-volume\") pod \"payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.425434 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.425406 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaafe556-aa91-411a-bc1b-2baac76e8b40-plugins-config-volume\") pod \"payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.505705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.485567 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrsk\" (UniqueName: \"kubernetes.io/projected/eaafe556-aa91-411a-bc1b-2baac76e8b40-kube-api-access-rhrsk\") pod \"payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.505705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.488773 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:23.682274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.682246 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:23.926742 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:23.926710 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaafe556_aa91_411a_bc1b_2baac76e8b40.slice/crio-56f3ee3c175c18e57c89385c65603717ef047eac6a21b12c466da5748399ff9e WatchSource:0}: Error finding container 56f3ee3c175c18e57c89385c65603717ef047eac6a21b12c466da5748399ff9e: Status 404 returned error can't find the container with id 56f3ee3c175c18e57c89385c65603717ef047eac6a21b12c466da5748399ff9e Oct 30 22:05:23.929398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:23.929377 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m"] Oct 30 22:05:24.077018 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:24.076992 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef410b18_912a_4d5f_a511_5115972a418b.slice/crio-58d163d3c777901f7a8659045092af9391f1e0c8702327de619c3acbeec6b336 WatchSource:0}: Error finding container 58d163d3c777901f7a8659045092af9391f1e0c8702327de619c3acbeec6b336: Status 404 returned error can't find the container with id 58d163d3c777901f7a8659045092af9391f1e0c8702327de619c3acbeec6b336 Oct 30 22:05:24.080393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.080366 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl"] Oct 30 22:05:24.258994 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.258913 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" event={"ID":"ef410b18-912a-4d5f-a511-5115972a418b","Type":"ContainerStarted","Data":"aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83"} Oct 30 22:05:24.258994 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.258971 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" event={"ID":"ef410b18-912a-4d5f-a511-5115972a418b","Type":"ContainerStarted","Data":"58d163d3c777901f7a8659045092af9391f1e0c8702327de619c3acbeec6b336"} Oct 30 22:05:24.259196 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.259152 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:24.261204 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.261177 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" event={"ID":"eaafe556-aa91-411a-bc1b-2baac76e8b40","Type":"ContainerStarted","Data":"0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249"} Oct 30 22:05:24.261365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.261220 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" event={"ID":"eaafe556-aa91-411a-bc1b-2baac76e8b40","Type":"ContainerStarted","Data":"56f3ee3c175c18e57c89385c65603717ef047eac6a21b12c466da5748399ff9e"} Oct 30 22:05:24.261365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.261267 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:24.383242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.383194 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" podStartSLOduration=2.383179177 podStartE2EDuration="2.383179177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:24.382008057 +0000 UTC m=+1373.063157952" watchObservedRunningTime="2026-10-30 22:05:24.383179177 +0000 UTC m=+1373.064329089" Oct 30 22:05:24.523247 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.523013 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" podStartSLOduration=1.522993859 podStartE2EDuration="1.522993859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:24.521653001 +0000 UTC m=+1373.202802894" watchObservedRunningTime="2026-10-30 22:05:24.522993859 +0000 UTC m=+1373.204143756" Oct 30 22:05:24.923504 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.923455 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs"] Oct 30 22:05:24.928133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:24.928114 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.040960 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.040928 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs"] Oct 30 22:05:25.051160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051131 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051297 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051334 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051635 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051612 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051697 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051657 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051721 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051859 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051899 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hwh\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-kube-api-access-q8hwh\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.051957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.051940 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157239 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157287 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157346 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157412 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157439 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hwh\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-kube-api-access-q8hwh\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157472 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157553 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157659 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.157690 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.158313 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.158707 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-credential-socket\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.158792 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istiod-ca-cert\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.158892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.158835 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-certs\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.161404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.159003 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-data\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.161404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.161163 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-envoy\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.162066 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.162037 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-podinfo\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.215201 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.215133 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-token\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.215386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.215356 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hwh\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-kube-api-access-q8hwh\") pod \"e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.244425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.244395 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:25.684460 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:25.684426 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbcc83d_9e4a_4dda_9936_7b0b1f3bbe97.slice/crio-b6b9f0b69358eca0725a1003ac8b2dd5bcf5b447184e39460f5fd0ad96cb2e22 WatchSource:0}: Error finding container b6b9f0b69358eca0725a1003ac8b2dd5bcf5b447184e39460f5fd0ad96cb2e22: Status 404 returned error can't find the container with id b6b9f0b69358eca0725a1003ac8b2dd5bcf5b447184e39460f5fd0ad96cb2e22 Oct 30 22:05:25.687257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.687220 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:25.687348 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.687291 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:25.687348 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.687319 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:25.688374 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:25.688348 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs"] Oct 30 22:05:26.292631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:26.292590 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" event={"ID":"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97","Type":"ContainerStarted","Data":"f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4"} Oct 30 22:05:26.293145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:26.292691 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" event={"ID":"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97","Type":"ContainerStarted","Data":"b6b9f0b69358eca0725a1003ac8b2dd5bcf5b447184e39460f5fd0ad96cb2e22"} Oct 30 22:05:26.421652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:26.421587 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podStartSLOduration=2.42156908 podStartE2EDuration="2.42156908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:26.415742691 +0000 UTC m=+1375.096892584" watchObservedRunningTime="2026-10-30 22:05:26.42156908 +0000 UTC m=+1375.102718974" Oct 30 22:05:27.253516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:27.250636 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:27.260667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:27.260636 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:27.260828 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:27.260705 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:28.017923 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.017756 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv"] Oct 30 22:05:28.090625 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.087904 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.096213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.096183 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv"] Oct 30 22:05:28.125404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.125359 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mfc\" (UniqueName: \"kubernetes.io/projected/d3222f4f-7d12-4ba8-943b-7bedc2102107-kube-api-access-s6mfc\") pod \"payload-processing-e2e-ait-97895160-675cbbd785-jzvkv\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.125628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.125528 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3222f4f-7d12-4ba8-943b-7bedc2102107-plugins-config-volume\") pod \"payload-processing-e2e-ait-97895160-675cbbd785-jzvkv\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.210760 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.210725 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-97895160\"" Oct 30 22:05:28.210944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.210837 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-97895160-dockercfg-qxxlw\"" Oct 30 22:05:28.227131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.227094 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mfc\" (UniqueName: \"kubernetes.io/projected/d3222f4f-7d12-4ba8-943b-7bedc2102107-kube-api-access-s6mfc\") pod \"payload-processing-e2e-ait-97895160-675cbbd785-jzvkv\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.227314 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.227277 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3222f4f-7d12-4ba8-943b-7bedc2102107-plugins-config-volume\") pod \"payload-processing-e2e-ait-97895160-675cbbd785-jzvkv\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.228559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.228530 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3222f4f-7d12-4ba8-943b-7bedc2102107-plugins-config-volume\") pod \"payload-processing-e2e-ait-97895160-675cbbd785-jzvkv\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.250521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.248404 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:28.250521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.248464 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:28.322983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.320010 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mfc\" (UniqueName: \"kubernetes.io/projected/d3222f4f-7d12-4ba8-943b-7bedc2102107-kube-api-access-s6mfc\") pod \"payload-processing-e2e-ait-97895160-675cbbd785-jzvkv\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.414025 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.412761 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:28.573656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.567657 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:05:28.625236 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:28.625185 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:05:28.987193 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:28.987161 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3222f4f_7d12_4ba8_943b_7bedc2102107.slice/crio-6d8fa68e86622ed9204143d739eb385fe0bdb528c051161e5af15c988b372998 WatchSource:0}: Error finding container 6d8fa68e86622ed9204143d739eb385fe0bdb528c051161e5af15c988b372998: Status 404 returned error can't find the container with id 6d8fa68e86622ed9204143d739eb385fe0bdb528c051161e5af15c988b372998 Oct 30 22:05:28.991541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:28.991474 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv"] Oct 30 22:05:29.249061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:29.248888 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:29.249061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:29.248949 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:29.379928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:29.379890 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" event={"ID":"d3222f4f-7d12-4ba8-943b-7bedc2102107","Type":"ContainerStarted","Data":"00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f"} Oct 30 22:05:29.380109 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:29.379967 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" event={"ID":"d3222f4f-7d12-4ba8-943b-7bedc2102107","Type":"ContainerStarted","Data":"6d8fa68e86622ed9204143d739eb385fe0bdb528c051161e5af15c988b372998"} Oct 30 22:05:29.380171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:29.380131 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:29.552513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:29.550858 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" podStartSLOduration=2.550835469 podStartE2EDuration="2.550835469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:29.525294273 +0000 UTC m=+1378.206444167" watchObservedRunningTime="2026-10-30 22:05:29.550835469 +0000 UTC m=+1378.231985363" Oct 30 22:05:30.231778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:30.231731 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:05:30.251516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:30.248870 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:30.251516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:30.248947 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:30.910573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:30.906753 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.169582 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.168639 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:05:31.203444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.202588 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.203792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.203760 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.245243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.244627 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:31.245243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.244687 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.245243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.244830 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:31.245243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.244860 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:31.577746 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.511694 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 22:05:31.577746 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.512117 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:31.698511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.692595 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:31.722547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.721632 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:05:31.722547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.722045 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:05:31.769514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.732691 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:05:31.876381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.876274 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:05:32.060512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:31.900328 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:05:32.060512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.003309 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:05:32.162460 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.162330 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:32.278654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.278587 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:05:32.393796 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.391912 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:32.870516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.858451 4933 scope.go:122] "RemoveContainer" containerID="c774f2035afc85d82e8094eaa27ef97e0ac5c945d1dbfbcefabd278e8115043f" Oct 30 22:05:32.940373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.940324 4933 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.332313716s: [/var/lib/containers/storage/overlay/0d0d251d3e8a48477089b14d98deb0c71a741d6173095a98d57fc8940ed820e8/diff /var/log/pods/openshift-multus_multus-vvgps_6a4f783d-df3d-4c74-939e-22bf246e9738/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 22:05:32.957700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.957646 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:05:32.965523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.963049 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:05:32.971268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.971215 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:05:33.133352 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.996653 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:05:33.469162 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.996737 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.469162 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:32.996761 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:05:33.469162 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.297276 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:05:33.469525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.297330 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:05:33.518971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.297351 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:05:33.543518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.336319 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:05:33.543518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.521386 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:33.543518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.521435 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:33.543518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.522115 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:33.543518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.522159 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:33.544228 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.543964 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:05:33.596516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.584037 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:33.655811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.652677 4933 scope.go:122] "RemoveContainer" containerID="1786b6d67ff49b3372432e42ce0dcc6dbcfdfce49cf9ecb4bcf6b1a1d3133c7c" Oct 30 22:05:33.974581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.972736 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:05:33.974581 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.972916 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:05:34.039892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.039849 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.103613 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.045460 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:34.128229 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:33.525454 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:05:34.159573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.156598 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.287619 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:34.264425 4933 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.628s" Oct 30 22:05:34.287619 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.264539 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:05:34.287619 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.264658 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:05:34.298233 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.298206 4933 scope.go:122] "RemoveContainer" containerID="1467edac65c36e199244b07748cbc99337c375d1d4f37e09679d394a554d9528" Oct 30 22:05:34.319929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.319139 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:34.319929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.319226 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:34.464659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.464628 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:05:34.554094 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.554067 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:05:34.556478 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.556456 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:05:34.556761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.556746 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:05:34.556900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.556887 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:05:34.557026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.557014 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:05:34.557149 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.557138 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:05:34.557269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.557258 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:05:34.557389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.557378 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:05:34.557542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.557529 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:05:34.557682 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.557670 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:05:34.631518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.611657 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.669409 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.621679 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:05:34.698519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.621725 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:05:34.724511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.621743 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:05:34.725076 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.621760 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:05:34.725483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.621775 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:05:34.725914 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.622675 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.726305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.622701 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.978740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.978630 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:34.979220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:34.979025 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:05:35.110544 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.104766 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.110544 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.104870 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.184570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.169869 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.184570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.170886 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.184570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.171628 4933 scope.go:122] "RemoveContainer" containerID="2cb86c0812988580e3ec53056cc3166b868434ae468bbb1b0be38331e2664d39" Oct 30 22:05:35.185233 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.185183 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:35.208704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.208647 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.256928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.256577 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:35.256928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.256630 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:35.262531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.258569 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:35.262531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.258619 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.438796 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.437844 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:05:35.733519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.716049 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:05:35.782935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.752610 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:05:35.783465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.752684 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:35.783872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.752703 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:05:35.784231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.752726 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:05:35.790311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.752743 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:05:35.790779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.752763 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:05:35.989576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:35.752796 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.032331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.032287 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.097303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.072296 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:05:36.121466 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.120704 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.136529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.127581 4933 patch_prober.go:29] interesting pod/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:36.136529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.127638 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.173042 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.173003 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.196213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.194267 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.412017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.411973 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:36.412323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.412273 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:36.423600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.423564 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.538905 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:36.538867 4933 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.078s" Oct 30 22:05:36.539312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.539290 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:05:36.627701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.627671 4933 scope.go:122] "RemoveContainer" containerID="9802b88fd15bea20641ee04b387b7dadfbff00b2bf1c94c5ca756f22cb45f390" Oct 30 22:05:36.788268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.788232 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.797006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.796871 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.815933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.815797 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 22:05:36.824724 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.822196 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:05:36.824724 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.822445 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:05:36.849829 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.842103 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:36.879696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:36.876155 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:05:37.084532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.069004 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.084532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.070128 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:05:37.114741 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.086560 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.128118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.127941 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:37.243682 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.231646 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:05:37.267785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.244111 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.293542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.268794 4933 scope.go:122] "RemoveContainer" containerID="5bc4b04d444f3da852cc2ddd845c7d6af64062b54e8a961f8fc4d7103b88e91c" Oct 30 22:05:37.293542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.268959 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:37.293542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.269017 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:37.541601 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.518574 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.819600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.789605 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.819600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.790936 4933 scope.go:122] "RemoveContainer" containerID="6c3fbaffaaa3616c6cf1791b96183089e8ec97d77b6013724a0745da24b1ee34" Oct 30 22:05:37.882527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.879921 4933 scope.go:122] "RemoveContainer" containerID="6b352bff0528fbd7f504b91713cdeed67331031c74262a8706d3f8de62034ea5" Oct 30 22:05:37.886526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.884249 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.886526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.885582 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.900517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.896606 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.900517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.896878 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.900517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.897085 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:05:37.900517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.899080 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:05:37.900517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.899683 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Oct 30 22:05:37.914699 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.914653 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:37.961996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.941876 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 30 22:05:37.999521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.995205 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:05:37.999521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.998203 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:37.999521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:37.998256 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.071118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.071027 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:05:38.072278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.072246 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.167131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.167089 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:05:38.238511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.237907 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.271417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.251177 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.329570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.324861 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 22:05:38.329570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.324924 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 22:05:38.408834 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.408807 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:05:38.542566 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.528465 4933 scope.go:122] "RemoveContainer" containerID="69ae64355267197b77e089cf7b03845b5f08f22f2136974aabcdb22ffa794e0c" Oct 30 22:05:38.862448 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.862418 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:05:38.878649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.878577 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:05:38.917738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.917663 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:05:38.929566 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.929195 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.930039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.930005 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:38.981669 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.980099 4933 scope.go:122] "RemoveContainer" containerID="ca2cdff939eb43dae6a445f50a190417e7ef7a6ae3af46524bc1fd1d70c40ff2" Oct 30 22:05:38.983506 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.982193 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:05:38.999023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:38.999001 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:05:39.033556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.031401 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:05:39.076118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.076073 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:39.084019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.079582 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:05:39.084259 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.084243 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:05:39.126813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.126790 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:05:39.127025 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.127013 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:05:39.127144 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.127135 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:05:39.176852 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.174422 4933 scope.go:122] "RemoveContainer" containerID="61563de3a85c34a1dec16a44a65e858871965f1ab0572cb4630dd3851434cf04" Oct 30 22:05:39.209891 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:39.209867 4933 scope.go:122] "RemoveContainer" containerID="42f190def8a6afd49645eba99b1eed472686e72cb86e7bc2b89293fa45527604" Oct 30 22:05:40.249093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:40.249066 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:40.321230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:40.320785 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:40.321230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:40.320848 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:40.735514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:40.734715 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:05:40.973509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:40.972684 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:41.243283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.240670 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:05:41.704007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.703967 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf"] Oct 30 22:05:41.708398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.708374 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.767185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.767150 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf"] Oct 30 22:05:41.767409 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.767383 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:05:41.767409 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.767242 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:41.767668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.767302 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:41.767668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.767242 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:41.811522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.811473 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf481b7-c320-492e-8514-79162382584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.811741 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.811637 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkp8\" (UniqueName: \"kubernetes.io/projected/cdf481b7-c320-492e-8514-79162382584c-kube-api-access-dbkp8\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.811741 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.811677 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf481b7-c320-492e-8514-79162382584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.913639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.913608 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkp8\" (UniqueName: \"kubernetes.io/projected/cdf481b7-c320-492e-8514-79162382584c-kube-api-access-dbkp8\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.913849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.913652 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf481b7-c320-492e-8514-79162382584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.913954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.913930 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf481b7-c320-492e-8514-79162382584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.914391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.914364 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf481b7-c320-492e-8514-79162382584c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.916414 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.916389 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf481b7-c320-492e-8514-79162382584c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:41.972027 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:41.971949 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkp8\" (UniqueName: \"kubernetes.io/projected/cdf481b7-c320-492e-8514-79162382584c-kube-api-access-dbkp8\") pod \"maas-api-revoke-sub-6a1c00c0-g7bxf\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:42.026738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:42.026702 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:42.428454 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:42.428420 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf481b7_c320_492e_8514_79162382584c.slice/crio-e8f72928c03bbeaa4184276b6152bc0b8cc37b054dc98ce11f40731f48f56cb1 WatchSource:0}: Error finding container e8f72928c03bbeaa4184276b6152bc0b8cc37b054dc98ce11f40731f48f56cb1: Status 404 returned error can't find the container with id e8f72928c03bbeaa4184276b6152bc0b8cc37b054dc98ce11f40731f48f56cb1 Oct 30 22:05:42.430568 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:42.430424 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf"] Oct 30 22:05:43.243649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.243542 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" event={"ID":"cdf481b7-c320-492e-8514-79162382584c","Type":"ContainerStarted","Data":"ebe6b4fcf57af4ff0d6475f47c1cc1736168f85c83801dd826526a10354cfedd"} Oct 30 22:05:43.244111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.243696 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" event={"ID":"cdf481b7-c320-492e-8514-79162382584c","Type":"ContainerStarted","Data":"e8f72928c03bbeaa4184276b6152bc0b8cc37b054dc98ce11f40731f48f56cb1"} Oct 30 22:05:43.413122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.413074 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" podStartSLOduration=2.413061314 podStartE2EDuration="2.413061314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:43.362798337 +0000 UTC m=+1392.043948231" watchObservedRunningTime="2026-10-30 22:05:43.413061314 +0000 UTC m=+1392.094211224" Oct 30 22:05:43.732557 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.732522 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl"] Oct 30 22:05:43.741732 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.741693 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.795922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.795866 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl"] Oct 30 22:05:43.810397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.810361 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:05:43.810397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.810361 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:05:43.810624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.810521 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:05:43.810624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.810572 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5jfgf\"" Oct 30 22:05:43.852007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.851967 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.852189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.852061 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.852283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.852263 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.852343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.852296 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.852422 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.852405 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.852477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.852441 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85ztl\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-kube-api-access-85ztl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.852704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.852685 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50246925-1aa8-4563-a2eb-47a1cd64c692-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.954110 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.954077 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.954110 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.954120 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.954773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.954222 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.954773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.954254 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85ztl\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-kube-api-access-85ztl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.954773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.954335 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50246925-1aa8-4563-a2eb-47a1cd64c692-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.954773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.954376 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.954773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.954455 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.955089 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.955044 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.958465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.958437 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50246925-1aa8-4563-a2eb-47a1cd64c692-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.958938 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.958913 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.959009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.958954 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:43.959357 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:43.959334 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:44.011895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:44.011864 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:44.012119 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:44.012097 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85ztl\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-kube-api-access-85ztl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9z7tl\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:44.059153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:44.059062 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:44.259341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:44.259305 4933 generic.go:356] "Generic (PLEG): container finished" podID="cdf481b7-c320-492e-8514-79162382584c" containerID="ebe6b4fcf57af4ff0d6475f47c1cc1736168f85c83801dd826526a10354cfedd" exitCode=0 Oct 30 22:05:44.259867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:44.259614 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" event={"ID":"cdf481b7-c320-492e-8514-79162382584c","Type":"ContainerDied","Data":"ebe6b4fcf57af4ff0d6475f47c1cc1736168f85c83801dd826526a10354cfedd"} Oct 30 22:05:44.544435 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:05:44.544388 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50246925_1aa8_4563_a2eb_47a1cd64c692.slice/crio-7ef1da8b3b763279a3b3048ddd05abf1d1bb1a17b389dd463d7264facff3dd80 WatchSource:0}: Error finding container 7ef1da8b3b763279a3b3048ddd05abf1d1bb1a17b389dd463d7264facff3dd80: Status 404 returned error can't find the container with id 7ef1da8b3b763279a3b3048ddd05abf1d1bb1a17b389dd463d7264facff3dd80 Oct 30 22:05:44.546345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:44.546318 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl"] Oct 30 22:05:45.272310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.272274 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" event={"ID":"50246925-1aa8-4563-a2eb-47a1cd64c692","Type":"ContainerStarted","Data":"7ef1da8b3b763279a3b3048ddd05abf1d1bb1a17b389dd463d7264facff3dd80"} Oct 30 22:05:45.473089 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.473063 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:45.579762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.579734 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf481b7-c320-492e-8514-79162382584c-kube-api-access-dbkp8\" (UniqueName: \"kubernetes.io/projected/cdf481b7-c320-492e-8514-79162382584c-kube-api-access-dbkp8\") pod \"cdf481b7-c320-492e-8514-79162382584c\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " Oct 30 22:05:45.579948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.579780 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf481b7-c320-492e-8514-79162382584c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf481b7-c320-492e-8514-79162382584c-maas-api-openshift-ca\") pod \"cdf481b7-c320-492e-8514-79162382584c\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " Oct 30 22:05:45.579948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.579926 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdf481b7-c320-492e-8514-79162382584c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf481b7-c320-492e-8514-79162382584c-maas-api-xks-ca\") pod \"cdf481b7-c320-492e-8514-79162382584c\" (UID: \"cdf481b7-c320-492e-8514-79162382584c\") " Oct 30 22:05:45.580188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.580163 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf481b7-c320-492e-8514-79162382584c-maas-api-openshift-ca" pod "cdf481b7-c320-492e-8514-79162382584c" (UID: "cdf481b7-c320-492e-8514-79162382584c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:45.580718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.580692 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf481b7-c320-492e-8514-79162382584c-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:45.582860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.582838 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf481b7-c320-492e-8514-79162382584c-maas-api-xks-ca" pod "cdf481b7-c320-492e-8514-79162382584c" (UID: "cdf481b7-c320-492e-8514-79162382584c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:45.582990 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.582967 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf481b7-c320-492e-8514-79162382584c-kube-api-access-dbkp8" pod "cdf481b7-c320-492e-8514-79162382584c" (UID: "cdf481b7-c320-492e-8514-79162382584c"). InnerVolumeSpecName "kube-api-access-dbkp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:45.682057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.682031 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf481b7-c320-492e-8514-79162382584c-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:45.682057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:45.682057 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbkp8\" (UniqueName: \"kubernetes.io/projected/cdf481b7-c320-492e-8514-79162382584c-kube-api-access-dbkp8\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:05:46.287027 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:46.286996 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" event={"ID":"cdf481b7-c320-492e-8514-79162382584c","Type":"ContainerDied","Data":"e8f72928c03bbeaa4184276b6152bc0b8cc37b054dc98ce11f40731f48f56cb1"} Oct 30 22:05:46.287509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:46.287029 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf" Oct 30 22:05:46.287509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:46.287053 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8f72928c03bbeaa4184276b6152bc0b8cc37b054dc98ce11f40731f48f56cb1" Oct 30 22:05:48.221555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:48.221511 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:48.221895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:48.221576 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:05:49.324102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:49.324057 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" event={"ID":"50246925-1aa8-4563-a2eb-47a1cd64c692","Type":"ContainerStarted","Data":"5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e"} Oct 30 22:05:49.324557 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:49.324131 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:49.326032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:49.326003 4933 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9z7tl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:05:49.326176 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:49.326054 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" podUID="50246925-1aa8-4563-a2eb-47a1cd64c692" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:05:49.537366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:49.537305 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" podStartSLOduration=2.8643976 podStartE2EDuration="6.53728873s" totalImagesPullingTime="3.67289113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:49.495046802 +0000 UTC m=+1398.176196695" watchObservedRunningTime="2026-10-30 22:05:49.53728873 +0000 UTC m=+1398.218438621" Oct 30 22:05:50.333421 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:05:50.333391 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:05:58.546972 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:05:58.546932 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:06:06.276722 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.276683 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928"] Oct 30 22:06:06.277753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.277735 4933 cpu_manager.go:414] "containerMap: removing container" podUID="cdf481b7-c320-492e-8514-79162382584c" containerName="revoke-keys" Oct 30 22:06:06.277811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.277756 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf481b7-c320-492e-8514-79162382584c" containerName="revoke-keys" Oct 30 22:06:06.278016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.278004 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf481b7-c320-492e-8514-79162382584c" containerName="revoke-keys" Oct 30 22:06:06.286983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.286958 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.377365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.377333 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928"] Oct 30 22:06:06.394621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.394587 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.394759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.394635 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.394759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.394732 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5021c0-a209-41c7-8416-f72d83edc4a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.394898 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.394878 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.394949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.394935 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.395125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.395105 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.395173 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.395158 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.395280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.395260 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8488\" (UniqueName: \"kubernetes.io/projected/3a5021c0-a209-41c7-8416-f72d83edc4a1-kube-api-access-v8488\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.395348 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.395306 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496625 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496663 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8488\" (UniqueName: \"kubernetes.io/projected/3a5021c0-a209-41c7-8416-f72d83edc4a1-kube-api-access-v8488\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496688 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496810 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496830 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496850 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5021c0-a209-41c7-8416-f72d83edc4a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496935 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.496983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496980 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.497202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.496994 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.497202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.497081 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.497202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.497088 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.497202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.497145 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.497415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.497246 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.497894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.497856 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5021c0-a209-41c7-8416-f72d83edc4a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.499393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.499376 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.499573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.499556 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.557702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.557647 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5021c0-a209-41c7-8416-f72d83edc4a1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.561641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.561621 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8488\" (UniqueName: \"kubernetes.io/projected/3a5021c0-a209-41c7-8416-f72d83edc4a1-kube-api-access-v8488\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4s928\" (UID: \"3a5021c0-a209-41c7-8416-f72d83edc4a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:06.607606 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:06.607580 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:07.045180 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:06:07.045145 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5021c0_a209_41c7_8416_f72d83edc4a1.slice/crio-7c9ef00de28988fca81ec8f728582e7c65048a0dd2871d4c67f41a0270510868 WatchSource:0}: Error finding container 7c9ef00de28988fca81ec8f728582e7c65048a0dd2871d4c67f41a0270510868: Status 404 returned error can't find the container with id 7c9ef00de28988fca81ec8f728582e7c65048a0dd2871d4c67f41a0270510868 Oct 30 22:06:07.047574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.047542 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:07.047680 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.047608 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:07.047680 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.047640 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:06:07.048284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.048259 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928"] Oct 30 22:06:07.484270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.484178 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" event={"ID":"3a5021c0-a209-41c7-8416-f72d83edc4a1","Type":"ContainerStarted","Data":"53a005844468c6f25a8f2c07e9c9cecd10f5527a171609501242f2a1602c377e"} Oct 30 22:06:07.484270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.484225 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" event={"ID":"3a5021c0-a209-41c7-8416-f72d83edc4a1","Type":"ContainerStarted","Data":"7c9ef00de28988fca81ec8f728582e7c65048a0dd2871d4c67f41a0270510868"} Oct 30 22:06:07.608391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.608358 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:07.615423 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:07.615356 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podStartSLOduration=1.615338645 podStartE2EDuration="1.615338645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:07.607520787 +0000 UTC m=+1416.288670681" watchObservedRunningTime="2026-10-30 22:06:07.615338645 +0000 UTC m=+1416.296488541" Oct 30 22:06:08.609982 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:08.609945 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 22:06:08.610384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:08.609995 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 22:06:09.608836 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:09.608808 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 22:06:09.608998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:09.608854 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 22:06:10.608546 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:10.608473 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 22:06:10.608963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:10.608573 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 22:06:11.608312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:11.608274 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 22:06:11.608551 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:11.608330 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 22:06:12.608552 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:12.608508 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 22:06:12.608938 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:12.608573 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 22:06:13.608751 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:13.608717 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 22:06:13.609143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:13.608785 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 22:06:14.607940 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:14.607897 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 22:06:14.608181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:14.607950 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 22:06:15.813996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:15.813966 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:15.815438 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:15.815411 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:15.815997 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:15.815975 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" Oct 30 22:06:19.199802 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.199750 4933 patch_prober.go:29] interesting pod/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:19.200438 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.200396 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.245639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.245591 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.246253 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.246226 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:06:19.673690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.673649 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:06:19.791518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.767955 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:06:19.862624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.862580 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:06:19.904635 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.904590 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:19.942558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.940589 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:06:20.001533 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:19.995577 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Oct 30 22:06:20.110516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.058863 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:06:20.110516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.059992 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.115514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.111872 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 22:06:20.196670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.159471 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:06:20.258711 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.245982 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:06:20.258711 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.246536 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:06:20.258711 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.248237 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:06:20.362517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.285802 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:06:20.696518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.671578 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.776545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.690823 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:06:20.777061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.777035 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:06:20.792711 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.762822 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:06:20.792711 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.733206 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.808512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.794001 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:06:20.808512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.800152 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:06:20.893609 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.893567 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:06:20.897921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.897895 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:06:20.912643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.911563 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.926678 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.926638 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:06:20.958582 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.942965 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:20.976783 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:20.976752 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:06:21.017879 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.016582 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.019561 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.018341 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:06:21.408444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.368842 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Oct 30 22:06:21.473835 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.472376 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:06:21.623532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.608121 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:06:21.643106 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.643052 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:21.715572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.687122 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 22:06:21.839517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.817649 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 22:06:21.839517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.818773 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 22:06:21.839517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.822341 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:06:21.839517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.822619 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:06:21.839517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.822870 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:06:21.839517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:21.823100 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:06:22.253975 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.133050 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 22:06:23.154220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.169550 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.259662 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.170429 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Oct 30 22:06:23.404552 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.228285 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.406222 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.228771 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 22:06:23.406618 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.391813 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.410132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.391867 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:23.410301 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.410273 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.410460 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.516877 4933 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.410608 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.550286 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 22:06:23.411777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.392331 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.412959 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.995872 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:06:23.413118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.413093 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.413235 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.846549 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.414249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.839612 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:06:23.415402 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.609093 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 22:06:23.416593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.846467 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 22:06:23.417846 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:22.968015 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 22:06:23.534570 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:06:23.523987 4933 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.677s" Oct 30 22:06:23.534570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.524068 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:06:23.846742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.834977 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:06:23.862028 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.861884 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:06:23.862704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.862053 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:06:23.862704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.862195 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:06:23.974998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:23.963699 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:06:24.076537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:24.072034 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:06:24.088880 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:24.078573 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:06:24.155460 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:24.155350 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:06:24.156017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:24.155466 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:06:24.306317 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:24.306281 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:06:24.309631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:24.309608 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:06:28.553128 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:06:28.553085 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:06:39.636164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:39.636130 4933 scope.go:122] "RemoveContainer" containerID="13683740bf0ae5395024b8e873deec52fc548884da23223bb6f339cf99141394" Oct 30 22:06:39.647242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:06:39.647221 4933 scope.go:122] "RemoveContainer" containerID="f08a76d36aed49b73e976f62b46bb0deada9f4e5a2dfe756e941526681a04503" Oct 30 22:06:58.547451 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:06:58.547409 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:07:00.150279 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:00.150241 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:07:06.050192 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:06.050154 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb"] Oct 30 22:07:06.104236 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:06.104202 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ca3806-vvtdb"] Oct 30 22:07:07.205989 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.205956 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2"] Oct 30 22:07:07.212519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.212215 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.269848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.269821 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2"] Oct 30 22:07:07.312705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.312666 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:07.313006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.312980 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:07:07.313075 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.313016 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:07.316397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.316377 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:07.326118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.326093 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.326296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.326273 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.326380 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.326371 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dlcc\" (UniqueName: \"kubernetes.io/projected/1af6cbe0-852a-492f-a2c9-59213fb8e80f-kube-api-access-4dlcc\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.428134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.428099 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.428256 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.428231 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.428298 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.428292 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dlcc\" (UniqueName: \"kubernetes.io/projected/1af6cbe0-852a-492f-a2c9-59213fb8e80f-kube-api-access-4dlcc\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.428912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.428891 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.430756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.430732 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.469213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.469141 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7decc5-01c0-4583-b540-b5ab1461d0d1" path="/var/lib/kubelet/pods/bc7decc5-01c0-4583-b540-b5ab1461d0d1/volumes" Oct 30 22:07:07.486631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.486610 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dlcc\" (UniqueName: \"kubernetes.io/projected/1af6cbe0-852a-492f-a2c9-59213fb8e80f-kube-api-access-4dlcc\") pod \"maas-api-revoke-keys-e2e-ait-1d71f464-khfh2\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.534700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.534682 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:07.956216 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:07:07.956186 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af6cbe0_852a_492f_a2c9_59213fb8e80f.slice/crio-1d5b1502a55868da0144a979100ed19718b8f8162b664b7f524215d407ac0ca2 WatchSource:0}: Error finding container 1d5b1502a55868da0144a979100ed19718b8f8162b664b7f524215d407ac0ca2: Status 404 returned error can't find the container with id 1d5b1502a55868da0144a979100ed19718b8f8162b664b7f524215d407ac0ca2 Oct 30 22:07:07.957979 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:07.957951 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2"] Oct 30 22:07:08.866460 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:08.866430 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" event={"ID":"1af6cbe0-852a-492f-a2c9-59213fb8e80f","Type":"ContainerStarted","Data":"d00fa0818f8886baf20b0e5885ab1745a097656772118b3e9eb71dc730db4c2f"} Oct 30 22:07:08.866912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:08.866501 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" event={"ID":"1af6cbe0-852a-492f-a2c9-59213fb8e80f","Type":"ContainerStarted","Data":"1d5b1502a55868da0144a979100ed19718b8f8162b664b7f524215d407ac0ca2"} Oct 30 22:07:09.028230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:09.028183 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" podStartSLOduration=2.028168518 podStartE2EDuration="2.028168518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:09.023570049 +0000 UTC m=+1477.704720145" watchObservedRunningTime="2026-10-30 22:07:09.028168518 +0000 UTC m=+1477.709318468" Oct 30 22:07:09.876595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:09.876563 4933 generic.go:356] "Generic (PLEG): container finished" podID="1af6cbe0-852a-492f-a2c9-59213fb8e80f" containerID="d00fa0818f8886baf20b0e5885ab1745a097656772118b3e9eb71dc730db4c2f" exitCode=0 Oct 30 22:07:09.876991 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:09.876653 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" event={"ID":"1af6cbe0-852a-492f-a2c9-59213fb8e80f","Type":"ContainerDied","Data":"d00fa0818f8886baf20b0e5885ab1745a097656772118b3e9eb71dc730db4c2f"} Oct 30 22:07:11.098976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.098949 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:11.288217 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.288183 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af6cbe0-852a-492f-a2c9-59213fb8e80f-kube-api-access-4dlcc\" (UniqueName: \"kubernetes.io/projected/1af6cbe0-852a-492f-a2c9-59213fb8e80f-kube-api-access-4dlcc\") pod \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " Oct 30 22:07:11.288425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.288345 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-xks-ca\") pod \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " Oct 30 22:07:11.288547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.288484 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-openshift-ca\") pod \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\" (UID: \"1af6cbe0-852a-492f-a2c9-59213fb8e80f\") " Oct 30 22:07:11.288884 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.288852 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-openshift-ca" pod "1af6cbe0-852a-492f-a2c9-59213fb8e80f" (UID: "1af6cbe0-852a-492f-a2c9-59213fb8e80f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:11.289153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.289061 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:11.290816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.290785 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-xks-ca" pod "1af6cbe0-852a-492f-a2c9-59213fb8e80f" (UID: "1af6cbe0-852a-492f-a2c9-59213fb8e80f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:11.290921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.290837 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af6cbe0-852a-492f-a2c9-59213fb8e80f-kube-api-access-4dlcc" pod "1af6cbe0-852a-492f-a2c9-59213fb8e80f" (UID: "1af6cbe0-852a-492f-a2c9-59213fb8e80f"). InnerVolumeSpecName "kube-api-access-4dlcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:11.390977 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.390952 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dlcc\" (UniqueName: \"kubernetes.io/projected/1af6cbe0-852a-492f-a2c9-59213fb8e80f-kube-api-access-4dlcc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:11.390977 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.390975 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af6cbe0-852a-492f-a2c9-59213fb8e80f-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:11.903834 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.903801 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" Oct 30 22:07:11.903834 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.903826 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2" event={"ID":"1af6cbe0-852a-492f-a2c9-59213fb8e80f","Type":"ContainerDied","Data":"1d5b1502a55868da0144a979100ed19718b8f8162b664b7f524215d407ac0ca2"} Oct 30 22:07:11.904037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:11.903873 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5b1502a55868da0144a979100ed19718b8f8162b664b7f524215d407ac0ca2" Oct 30 22:07:17.557453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.557421 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x"] Oct 30 22:07:17.558126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.558111 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1af6cbe0-852a-492f-a2c9-59213fb8e80f" containerName="revoke-keys" Oct 30 22:07:17.558174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.558127 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af6cbe0-852a-492f-a2c9-59213fb8e80f" containerName="revoke-keys" Oct 30 22:07:17.558256 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.558245 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af6cbe0-852a-492f-a2c9-59213fb8e80f" containerName="revoke-keys" Oct 30 22:07:17.562252 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.562237 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.579585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.579556 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.579709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.579615 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.579808 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.579785 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rht6b\" (UniqueName: \"kubernetes.io/projected/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-kube-api-access-rht6b\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.615235 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.615202 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:17.615453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.615435 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:17.615555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.615473 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:17.615555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.615471 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:07:17.618377 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.618352 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x"] Oct 30 22:07:17.681595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.681567 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.681746 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.681655 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.681814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.681779 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rht6b\" (UniqueName: \"kubernetes.io/projected/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-kube-api-access-rht6b\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.682310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.682286 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.684166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.684139 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.748121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.748092 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rht6b\" (UniqueName: \"kubernetes.io/projected/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-kube-api-access-rht6b\") pod \"maas-api-revoke-keys-e2e-ait-64086213-hqc6x\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:17.876927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:17.876856 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:18.186479 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:18.186406 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk"] Oct 30 22:07:18.293231 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:07:18.293203 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4b7561_a31f_4b13_a6c2_34948b57c0ea.slice/crio-0c75b9ac5e7c6835d80a63ee3e9cc82aa8764d755afffff382d7797330a61eb6 WatchSource:0}: Error finding container 0c75b9ac5e7c6835d80a63ee3e9cc82aa8764d755afffff382d7797330a61eb6: Status 404 returned error can't find the container with id 0c75b9ac5e7c6835d80a63ee3e9cc82aa8764d755afffff382d7797330a61eb6 Oct 30 22:07:18.300634 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:18.300402 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x"] Oct 30 22:07:18.994424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:18.994321 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" event={"ID":"7d4b7561-a31f-4b13-a6c2-34948b57c0ea","Type":"ContainerStarted","Data":"bef74aafdc27c06c1711affd540887e5a4b88f6df0c386b0eaaa758f5a8c2adc"} Oct 30 22:07:18.994424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:18.994387 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" event={"ID":"7d4b7561-a31f-4b13-a6c2-34948b57c0ea","Type":"ContainerStarted","Data":"0c75b9ac5e7c6835d80a63ee3e9cc82aa8764d755afffff382d7797330a61eb6"} Oct 30 22:07:19.116864 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:19.116811 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" podStartSLOduration=2.116795043 podStartE2EDuration="2.116795043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:19.113478009 +0000 UTC m=+1487.794627902" watchObservedRunningTime="2026-10-30 22:07:19.116795043 +0000 UTC m=+1487.797944968" Oct 30 22:07:20.006129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:20.006095 4933 generic.go:356] "Generic (PLEG): container finished" podID="7d4b7561-a31f-4b13-a6c2-34948b57c0ea" containerID="bef74aafdc27c06c1711affd540887e5a4b88f6df0c386b0eaaa758f5a8c2adc" exitCode=0 Oct 30 22:07:20.006594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:20.006179 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" event={"ID":"7d4b7561-a31f-4b13-a6c2-34948b57c0ea","Type":"ContainerDied","Data":"bef74aafdc27c06c1711affd540887e5a4b88f6df0c386b0eaaa758f5a8c2adc"} Oct 30 22:07:21.201083 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.201061 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:21.306317 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.306289 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-kube-api-access-rht6b\" (UniqueName: \"kubernetes.io/projected/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-kube-api-access-rht6b\") pod \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " Oct 30 22:07:21.306536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.306358 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-xks-ca\") pod \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " Oct 30 22:07:21.306681 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.306664 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-openshift-ca\") pod \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\" (UID: \"7d4b7561-a31f-4b13-a6c2-34948b57c0ea\") " Oct 30 22:07:21.307027 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.307005 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-openshift-ca" pod "7d4b7561-a31f-4b13-a6c2-34948b57c0ea" (UID: "7d4b7561-a31f-4b13-a6c2-34948b57c0ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:21.307344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.307329 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:21.308719 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.308691 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-xks-ca" pod "7d4b7561-a31f-4b13-a6c2-34948b57c0ea" (UID: "7d4b7561-a31f-4b13-a6c2-34948b57c0ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:21.308830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.308739 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-kube-api-access-rht6b" pod "7d4b7561-a31f-4b13-a6c2-34948b57c0ea" (UID: "7d4b7561-a31f-4b13-a6c2-34948b57c0ea"). InnerVolumeSpecName "kube-api-access-rht6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:21.409292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.409253 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rht6b\" (UniqueName: \"kubernetes.io/projected/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-kube-api-access-rht6b\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:21.409292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:21.409284 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4b7561-a31f-4b13-a6c2-34948b57c0ea-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:22.034565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:22.034529 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" Oct 30 22:07:22.034771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:22.034524 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x" event={"ID":"7d4b7561-a31f-4b13-a6c2-34948b57c0ea","Type":"ContainerDied","Data":"0c75b9ac5e7c6835d80a63ee3e9cc82aa8764d755afffff382d7797330a61eb6"} Oct 30 22:07:22.034771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:22.034709 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c75b9ac5e7c6835d80a63ee3e9cc82aa8764d755afffff382d7797330a61eb6" Oct 30 22:07:23.138377 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.138345 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc"] Oct 30 22:07:23.139028 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.139014 4933 cpu_manager.go:414] "containerMap: removing container" podUID="7d4b7561-a31f-4b13-a6c2-34948b57c0ea" containerName="revoke-keys" Oct 30 22:07:23.139078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.139029 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4b7561-a31f-4b13-a6c2-34948b57c0ea" containerName="revoke-keys" Oct 30 22:07:23.139165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.139156 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4b7561-a31f-4b13-a6c2-34948b57c0ea" containerName="revoke-keys" Oct 30 22:07:23.143065 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.143047 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.187651 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.187612 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" podUID="0b6bf96d-06d8-4ea7-b59a-4e7659773409" containerName="payload-processing" containerID="cri-o://313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8" gracePeriod=25 Oct 30 22:07:23.201802 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.201779 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:23.205367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.205343 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc"] Oct 30 22:07:23.237587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.237078 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.237587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.237145 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.237587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.237186 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmphc\" (UniqueName: \"kubernetes.io/projected/2a1c882f-bb5d-4af2-b539-ff75939dea39-kube-api-access-kmphc\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.247321 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.247212 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:07:23.252647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.252598 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:23.298132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.298106 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:23.339335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.339297 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.339505 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.339364 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmphc\" (UniqueName: \"kubernetes.io/projected/2a1c882f-bb5d-4af2-b539-ff75939dea39-kube-api-access-kmphc\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.339812 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.339791 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.340713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.340683 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.343220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.343190 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.401357 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.401294 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmphc\" (UniqueName: \"kubernetes.io/projected/2a1c882f-bb5d-4af2-b539-ff75939dea39-kube-api-access-kmphc\") pod \"maas-api-revoke-sub-f9421245-wfhtc\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.461814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.461775 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:23.528099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.528070 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:07:23.643954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.643914 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6bf96d-06d8-4ea7-b59a-4e7659773409-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6bf96d-06d8-4ea7-b59a-4e7659773409-plugins-config-volume\") pod \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " Oct 30 22:07:23.644161 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.643978 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6bf96d-06d8-4ea7-b59a-4e7659773409-kube-api-access-7bd86\" (UniqueName: \"kubernetes.io/projected/0b6bf96d-06d8-4ea7-b59a-4e7659773409-kube-api-access-7bd86\") pod \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\" (UID: \"0b6bf96d-06d8-4ea7-b59a-4e7659773409\") " Oct 30 22:07:23.644698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.644663 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6bf96d-06d8-4ea7-b59a-4e7659773409-plugins-config-volume" pod "0b6bf96d-06d8-4ea7-b59a-4e7659773409" (UID: "0b6bf96d-06d8-4ea7-b59a-4e7659773409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:23.647250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.647227 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6bf96d-06d8-4ea7-b59a-4e7659773409-kube-api-access-7bd86" pod "0b6bf96d-06d8-4ea7-b59a-4e7659773409" (UID: "0b6bf96d-06d8-4ea7-b59a-4e7659773409"). InnerVolumeSpecName "kube-api-access-7bd86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:23.745845 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.745771 4933 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b6bf96d-06d8-4ea7-b59a-4e7659773409-plugins-config-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:23.745845 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.745798 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bd86\" (UniqueName: \"kubernetes.io/projected/0b6bf96d-06d8-4ea7-b59a-4e7659773409-kube-api-access-7bd86\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:23.862136 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:07:23.862106 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1c882f_bb5d_4af2_b539_ff75939dea39.slice/crio-a17657a84d123dc4de73200a99f3a53d98176c24d27f9ad1025a4308e0c9094c WatchSource:0}: Error finding container a17657a84d123dc4de73200a99f3a53d98176c24d27f9ad1025a4308e0c9094c: Status 404 returned error can't find the container with id a17657a84d123dc4de73200a99f3a53d98176c24d27f9ad1025a4308e0c9094c Oct 30 22:07:23.864562 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:23.864536 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc"] Oct 30 22:07:24.055826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.055794 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" event={"ID":"2a1c882f-bb5d-4af2-b539-ff75939dea39","Type":"ContainerStarted","Data":"a17657a84d123dc4de73200a99f3a53d98176c24d27f9ad1025a4308e0c9094c"} Oct 30 22:07:24.057560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.057530 4933 generic.go:356] "Generic (PLEG): container finished" podID="0b6bf96d-06d8-4ea7-b59a-4e7659773409" containerID="313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8" exitCode=0 Oct 30 22:07:24.057688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.057584 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" event={"ID":"0b6bf96d-06d8-4ea7-b59a-4e7659773409","Type":"ContainerDied","Data":"313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8"} Oct 30 22:07:24.057688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.057594 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" Oct 30 22:07:24.057688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.057623 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk" event={"ID":"0b6bf96d-06d8-4ea7-b59a-4e7659773409","Type":"ContainerDied","Data":"041d401372aca83e3b745c68dc11438dbde64dc99779c55681fb8da5f6f75653"} Oct 30 22:07:24.057688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.057646 4933 scope.go:122] "RemoveContainer" containerID="313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8" Oct 30 22:07:24.078198 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.078179 4933 scope.go:122] "RemoveContainer" containerID="313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8" Oct 30 22:07:24.078569 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:07:24.078500 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8\": container with ID starting with 313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8 not found: ID does not exist" containerID="313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8" Oct 30 22:07:24.078569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.078539 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8"} err="failed to get container status \"313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8\": rpc error: code = NotFound desc = could not find container \"313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8\": container with ID starting with 313610a96089e0e909bedc792ad83f47c4bcc4a4fcb5dbc8d3d872ca6f3519d8 not found: ID does not exist" Oct 30 22:07:24.228389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.228351 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk"] Oct 30 22:07:24.233758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:24.233734 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1d71f464-665fbbff49-zckbk"] Oct 30 22:07:25.070293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:25.070209 4933 generic.go:356] "Generic (PLEG): container finished" podID="2a1c882f-bb5d-4af2-b539-ff75939dea39" containerID="2a9b706b5b5230828fd05a5fe919f533586bf569a36fd927f467a8aaafa60214" exitCode=0 Oct 30 22:07:25.070293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:25.070259 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" event={"ID":"2a1c882f-bb5d-4af2-b539-ff75939dea39","Type":"ContainerDied","Data":"2a9b706b5b5230828fd05a5fe919f533586bf569a36fd927f467a8aaafa60214"} Oct 30 22:07:25.470731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:25.470652 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6bf96d-06d8-4ea7-b59a-4e7659773409" path="/var/lib/kubelet/pods/0b6bf96d-06d8-4ea7-b59a-4e7659773409/volumes" Oct 30 22:07:26.420058 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.420034 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:26.499707 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.499678 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-xks-ca\") pod \"2a1c882f-bb5d-4af2-b539-ff75939dea39\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " Oct 30 22:07:26.500112 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.499929 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-openshift-ca\") pod \"2a1c882f-bb5d-4af2-b539-ff75939dea39\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " Oct 30 22:07:26.500112 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.499992 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1c882f-bb5d-4af2-b539-ff75939dea39-kube-api-access-kmphc\" (UniqueName: \"kubernetes.io/projected/2a1c882f-bb5d-4af2-b539-ff75939dea39-kube-api-access-kmphc\") pod \"2a1c882f-bb5d-4af2-b539-ff75939dea39\" (UID: \"2a1c882f-bb5d-4af2-b539-ff75939dea39\") " Oct 30 22:07:26.500357 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.500322 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-openshift-ca" pod "2a1c882f-bb5d-4af2-b539-ff75939dea39" (UID: "2a1c882f-bb5d-4af2-b539-ff75939dea39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:26.500799 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.500775 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:26.502794 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.502765 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-xks-ca" pod "2a1c882f-bb5d-4af2-b539-ff75939dea39" (UID: "2a1c882f-bb5d-4af2-b539-ff75939dea39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:26.502890 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.502798 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1c882f-bb5d-4af2-b539-ff75939dea39-kube-api-access-kmphc" pod "2a1c882f-bb5d-4af2-b539-ff75939dea39" (UID: "2a1c882f-bb5d-4af2-b539-ff75939dea39"). InnerVolumeSpecName "kube-api-access-kmphc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:26.602453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.602415 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1c882f-bb5d-4af2-b539-ff75939dea39-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:26.602453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:26.602453 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmphc\" (UniqueName: \"kubernetes.io/projected/2a1c882f-bb5d-4af2-b539-ff75939dea39-kube-api-access-kmphc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:27.097120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:27.097092 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" Oct 30 22:07:27.097293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:27.097112 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc" event={"ID":"2a1c882f-bb5d-4af2-b539-ff75939dea39","Type":"ContainerDied","Data":"a17657a84d123dc4de73200a99f3a53d98176c24d27f9ad1025a4308e0c9094c"} Oct 30 22:07:27.097293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:27.097170 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17657a84d123dc4de73200a99f3a53d98176c24d27f9ad1025a4308e0c9094c" Oct 30 22:07:28.548729 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:07:28.548695 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:07:28.724069 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:28.724035 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m"] Oct 30 22:07:28.837803 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:28.837726 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl"] Oct 30 22:07:31.777172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:31.777143 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:07:31.780655 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:31.780630 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:07:32.501587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.501556 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc"] Oct 30 22:07:32.502243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.502229 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0b6bf96d-06d8-4ea7-b59a-4e7659773409" containerName="payload-processing" Oct 30 22:07:32.502296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.502245 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6bf96d-06d8-4ea7-b59a-4e7659773409" containerName="payload-processing" Oct 30 22:07:32.502296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.502274 4933 cpu_manager.go:414] "containerMap: removing container" podUID="2a1c882f-bb5d-4af2-b539-ff75939dea39" containerName="revoke-keys" Oct 30 22:07:32.502296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.502279 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1c882f-bb5d-4af2-b539-ff75939dea39" containerName="revoke-keys" Oct 30 22:07:32.502400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.502394 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1c882f-bb5d-4af2-b539-ff75939dea39" containerName="revoke-keys" Oct 30 22:07:32.502437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.502408 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6bf96d-06d8-4ea7-b59a-4e7659773409" containerName="payload-processing" Oct 30 22:07:32.505864 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.505845 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.566605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.566578 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc"] Oct 30 22:07:32.607904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.607878 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:07:32.608070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.607926 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:07:32.608070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.607930 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:07:32.614870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.614846 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:07:32.657413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.657384 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qhb\" (UniqueName: \"kubernetes.io/projected/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-kube-api-access-h6qhb\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.657590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.657460 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.657590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.657536 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.759841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.759747 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qhb\" (UniqueName: \"kubernetes.io/projected/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-kube-api-access-h6qhb\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.760001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.759845 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.760001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.759888 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.760456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.760436 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.762453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.762427 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.817726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.817695 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qhb\" (UniqueName: \"kubernetes.io/projected/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-kube-api-access-h6qhb\") pod \"maas-api-revoke-keys-e2e-ait-97895160-wfptc\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:32.821617 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:32.821598 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:33.221351 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:33.221300 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc"] Oct 30 22:07:33.228440 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:07:33.228405 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode250f7aa_1fb1_42a6_88a4_b6ee7953efe4.slice/crio-4d26ce983568e233633ffbd63deb04eb7812e1b462cc2b0f4953d4d86463d7cc WatchSource:0}: Error finding container 4d26ce983568e233633ffbd63deb04eb7812e1b462cc2b0f4953d4d86463d7cc: Status 404 returned error can't find the container with id 4d26ce983568e233633ffbd63deb04eb7812e1b462cc2b0f4953d4d86463d7cc Oct 30 22:07:33.725129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:33.725038 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" podUID="eaafe556-aa91-411a-bc1b-2baac76e8b40" containerName="payload-processing" containerID="cri-o://0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249" gracePeriod=25 Oct 30 22:07:33.838645 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:33.838595 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" podUID="ef410b18-912a-4d5f-a511-5115972a418b" containerName="payload-pre-processing" containerID="cri-o://aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83" gracePeriod=25 Oct 30 22:07:34.021218 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.021107 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:07:34.084509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.080612 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaafe556-aa91-411a-bc1b-2baac76e8b40-kube-api-access-rhrsk\" (UniqueName: \"kubernetes.io/projected/eaafe556-aa91-411a-bc1b-2baac76e8b40-kube-api-access-rhrsk\") pod \"eaafe556-aa91-411a-bc1b-2baac76e8b40\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " Oct 30 22:07:34.084509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.080792 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaafe556-aa91-411a-bc1b-2baac76e8b40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaafe556-aa91-411a-bc1b-2baac76e8b40-plugins-config-volume\") pod \"eaafe556-aa91-411a-bc1b-2baac76e8b40\" (UID: \"eaafe556-aa91-411a-bc1b-2baac76e8b40\") " Oct 30 22:07:34.084509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.082071 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaafe556-aa91-411a-bc1b-2baac76e8b40-plugins-config-volume" pod "eaafe556-aa91-411a-bc1b-2baac76e8b40" (UID: "eaafe556-aa91-411a-bc1b-2baac76e8b40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:34.096392 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.096361 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaafe556-aa91-411a-bc1b-2baac76e8b40-kube-api-access-rhrsk" pod "eaafe556-aa91-411a-bc1b-2baac76e8b40" (UID: "eaafe556-aa91-411a-bc1b-2baac76e8b40"). InnerVolumeSpecName "kube-api-access-rhrsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:34.131829 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.131807 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:07:34.165395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.165359 4933 generic.go:356] "Generic (PLEG): container finished" podID="ef410b18-912a-4d5f-a511-5115972a418b" containerID="aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83" exitCode=0 Oct 30 22:07:34.165600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.165412 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" Oct 30 22:07:34.165600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.165469 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" event={"ID":"ef410b18-912a-4d5f-a511-5115972a418b","Type":"ContainerDied","Data":"aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83"} Oct 30 22:07:34.165600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.165565 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl" event={"ID":"ef410b18-912a-4d5f-a511-5115972a418b","Type":"ContainerDied","Data":"58d163d3c777901f7a8659045092af9391f1e0c8702327de619c3acbeec6b336"} Oct 30 22:07:34.165600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.165594 4933 scope.go:122] "RemoveContainer" containerID="aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83" Oct 30 22:07:34.167381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.167357 4933 generic.go:356] "Generic (PLEG): container finished" podID="eaafe556-aa91-411a-bc1b-2baac76e8b40" containerID="0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249" exitCode=0 Oct 30 22:07:34.167485 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.167423 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" Oct 30 22:07:34.167591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.167556 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" event={"ID":"eaafe556-aa91-411a-bc1b-2baac76e8b40","Type":"ContainerDied","Data":"0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249"} Oct 30 22:07:34.167649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.167605 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m" event={"ID":"eaafe556-aa91-411a-bc1b-2baac76e8b40","Type":"ContainerDied","Data":"56f3ee3c175c18e57c89385c65603717ef047eac6a21b12c466da5748399ff9e"} Oct 30 22:07:34.171238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.171175 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" event={"ID":"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4","Type":"ContainerStarted","Data":"4bca01d9b264105f93f46034de36deb64cd6fe7c9c04babef9f21d5f2290bd62"} Oct 30 22:07:34.172120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.171239 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" event={"ID":"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4","Type":"ContainerStarted","Data":"4d26ce983568e233633ffbd63deb04eb7812e1b462cc2b0f4953d4d86463d7cc"} Oct 30 22:07:34.183687 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.183660 4933 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaafe556-aa91-411a-bc1b-2baac76e8b40-plugins-config-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:34.183800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.183694 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrsk\" (UniqueName: \"kubernetes.io/projected/eaafe556-aa91-411a-bc1b-2baac76e8b40-kube-api-access-rhrsk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:34.196963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.196924 4933 scope.go:122] "RemoveContainer" containerID="aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83" Oct 30 22:07:34.197531 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:07:34.197479 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83\": container with ID starting with aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83 not found: ID does not exist" containerID="aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83" Oct 30 22:07:34.197644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.197531 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83"} err="failed to get container status \"aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83\": rpc error: code = NotFound desc = could not find container \"aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83\": container with ID starting with aa600a107b8d1612ca05dfb35feb65a59e89023b39202c2555c16866131d6c83 not found: ID does not exist" Oct 30 22:07:34.197644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.197548 4933 scope.go:122] "RemoveContainer" containerID="0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249" Oct 30 22:07:34.212104 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.212079 4933 scope.go:122] "RemoveContainer" containerID="0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249" Oct 30 22:07:34.212418 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:07:34.212395 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249\": container with ID starting with 0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249 not found: ID does not exist" containerID="0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249" Oct 30 22:07:34.212535 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.212423 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249"} err="failed to get container status \"0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249\": rpc error: code = NotFound desc = could not find container \"0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249\": container with ID starting with 0e7e36897913e2b8ae6b83f48b05d30ae6d534f3f1a33fcd6f0f1d143c01f249 not found: ID does not exist" Oct 30 22:07:34.284800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.284773 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef410b18-912a-4d5f-a511-5115972a418b-kube-api-access-hcm47\" (UniqueName: \"kubernetes.io/projected/ef410b18-912a-4d5f-a511-5115972a418b-kube-api-access-hcm47\") pod \"ef410b18-912a-4d5f-a511-5115972a418b\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " Oct 30 22:07:34.284975 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.284962 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef410b18-912a-4d5f-a511-5115972a418b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef410b18-912a-4d5f-a511-5115972a418b-plugins-config-volume\") pod \"ef410b18-912a-4d5f-a511-5115972a418b\" (UID: \"ef410b18-912a-4d5f-a511-5115972a418b\") " Oct 30 22:07:34.285346 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.285308 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef410b18-912a-4d5f-a511-5115972a418b-plugins-config-volume" pod "ef410b18-912a-4d5f-a511-5115972a418b" (UID: "ef410b18-912a-4d5f-a511-5115972a418b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:34.285446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.285422 4933 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef410b18-912a-4d5f-a511-5115972a418b-plugins-config-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:34.286862 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.286838 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef410b18-912a-4d5f-a511-5115972a418b-kube-api-access-hcm47" pod "ef410b18-912a-4d5f-a511-5115972a418b" (UID: "ef410b18-912a-4d5f-a511-5115972a418b"). InnerVolumeSpecName "kube-api-access-hcm47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:34.386558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.386528 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcm47\" (UniqueName: \"kubernetes.io/projected/ef410b18-912a-4d5f-a511-5115972a418b-kube-api-access-hcm47\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:34.488897 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.488818 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m"] Oct 30 22:07:34.494214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.494194 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-64086213-67cb9567ff-t6w4m"] Oct 30 22:07:34.607537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.607468 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" podStartSLOduration=2.607455025 podStartE2EDuration="2.607455025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:34.601567348 +0000 UTC m=+1503.282717238" watchObservedRunningTime="2026-10-30 22:07:34.607455025 +0000 UTC m=+1503.288604952" Oct 30 22:07:34.719870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.719838 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl"] Oct 30 22:07:34.725209 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:34.725186 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-64086213-6bd8c7585f-xvncl"] Oct 30 22:07:35.182628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:35.182543 4933 generic.go:356] "Generic (PLEG): container finished" podID="e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" containerID="4bca01d9b264105f93f46034de36deb64cd6fe7c9c04babef9f21d5f2290bd62" exitCode=0 Oct 30 22:07:35.183029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:35.182624 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" event={"ID":"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4","Type":"ContainerDied","Data":"4bca01d9b264105f93f46034de36deb64cd6fe7c9c04babef9f21d5f2290bd62"} Oct 30 22:07:35.469703 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:35.469629 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaafe556-aa91-411a-bc1b-2baac76e8b40" path="/var/lib/kubelet/pods/eaafe556-aa91-411a-bc1b-2baac76e8b40/volumes" Oct 30 22:07:35.470045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:35.470030 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef410b18-912a-4d5f-a511-5115972a418b" path="/var/lib/kubelet/pods/ef410b18-912a-4d5f-a511-5115972a418b/volumes" Oct 30 22:07:36.376182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.376158 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:36.516029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.515951 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-openshift-ca\") pod \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " Oct 30 22:07:36.516204 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.516083 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-xks-ca\") pod \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " Oct 30 22:07:36.516327 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.516298 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-kube-api-access-h6qhb\" (UniqueName: \"kubernetes.io/projected/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-kube-api-access-h6qhb\") pod \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\" (UID: \"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4\") " Oct 30 22:07:36.516434 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.516388 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-openshift-ca" pod "e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" (UID: "e250f7aa-1fb1-42a6-88a4-b6ee7953efe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:36.516961 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.516935 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:36.518347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.518321 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-xks-ca" pod "e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" (UID: "e250f7aa-1fb1-42a6-88a4-b6ee7953efe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:07:36.518477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.518435 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-kube-api-access-h6qhb" pod "e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" (UID: "e250f7aa-1fb1-42a6-88a4-b6ee7953efe4"). InnerVolumeSpecName "kube-api-access-h6qhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:36.618300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.618271 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:36.618472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:36.618304 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6qhb\" (UniqueName: \"kubernetes.io/projected/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4-kube-api-access-h6qhb\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:37.204255 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:37.204228 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" Oct 30 22:07:37.204409 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:37.204225 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc" event={"ID":"e250f7aa-1fb1-42a6-88a4-b6ee7953efe4","Type":"ContainerDied","Data":"4d26ce983568e233633ffbd63deb04eb7812e1b462cc2b0f4953d4d86463d7cc"} Oct 30 22:07:37.204409 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:37.204344 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d26ce983568e233633ffbd63deb04eb7812e1b462cc2b0f4953d4d86463d7cc" Oct 30 22:07:38.051608 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:38.051577 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c"] Oct 30 22:07:38.101363 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:38.101330 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04f24c0e-bl52c"] Oct 30 22:07:39.469862 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:39.469832 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13960e0-0572-4d40-8c2e-41f6bfd5b877" path="/var/lib/kubelet/pods/b13960e0-0572-4d40-8c2e-41f6bfd5b877/volumes" Oct 30 22:07:43.957400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:43.957359 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv"] Oct 30 22:07:48.958450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:48.958406 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" podUID="d3222f4f-7d12-4ba8-943b-7bedc2102107" containerName="payload-processing" containerID="cri-o://00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f" gracePeriod=25 Oct 30 22:07:49.253152 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.253127 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:07:49.298951 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.298926 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3222f4f-7d12-4ba8-943b-7bedc2102107-kube-api-access-s6mfc\" (UniqueName: \"kubernetes.io/projected/d3222f4f-7d12-4ba8-943b-7bedc2102107-kube-api-access-s6mfc\") pod \"d3222f4f-7d12-4ba8-943b-7bedc2102107\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " Oct 30 22:07:49.299086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.298963 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3222f4f-7d12-4ba8-943b-7bedc2102107-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3222f4f-7d12-4ba8-943b-7bedc2102107-plugins-config-volume\") pod \"d3222f4f-7d12-4ba8-943b-7bedc2102107\" (UID: \"d3222f4f-7d12-4ba8-943b-7bedc2102107\") " Oct 30 22:07:49.299405 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.299379 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3222f4f-7d12-4ba8-943b-7bedc2102107-plugins-config-volume" pod "d3222f4f-7d12-4ba8-943b-7bedc2102107" (UID: "d3222f4f-7d12-4ba8-943b-7bedc2102107"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:49.299619 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.299597 4933 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3222f4f-7d12-4ba8-943b-7bedc2102107-plugins-config-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:49.301129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.301107 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3222f4f-7d12-4ba8-943b-7bedc2102107-kube-api-access-s6mfc" pod "d3222f4f-7d12-4ba8-943b-7bedc2102107" (UID: "d3222f4f-7d12-4ba8-943b-7bedc2102107"). InnerVolumeSpecName "kube-api-access-s6mfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:49.316657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.316631 4933 generic.go:356] "Generic (PLEG): container finished" podID="d3222f4f-7d12-4ba8-943b-7bedc2102107" containerID="00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f" exitCode=0 Oct 30 22:07:49.316766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.316679 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" Oct 30 22:07:49.316766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.316697 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" event={"ID":"d3222f4f-7d12-4ba8-943b-7bedc2102107","Type":"ContainerDied","Data":"00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f"} Oct 30 22:07:49.316766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.316748 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv" event={"ID":"d3222f4f-7d12-4ba8-943b-7bedc2102107","Type":"ContainerDied","Data":"6d8fa68e86622ed9204143d739eb385fe0bdb528c051161e5af15c988b372998"} Oct 30 22:07:49.316915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.316776 4933 scope.go:122] "RemoveContainer" containerID="00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f" Oct 30 22:07:49.329472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.329457 4933 scope.go:122] "RemoveContainer" containerID="00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f" Oct 30 22:07:49.329798 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:07:49.329781 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f\": container with ID starting with 00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f not found: ID does not exist" containerID="00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f" Oct 30 22:07:49.329853 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.329808 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f"} err="failed to get container status \"00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f\": rpc error: code = NotFound desc = could not find container \"00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f\": container with ID starting with 00bf1318f84463bd2b1ab616ecb8f5b4f37cf19b331299f088b682d9abb43e9f not found: ID does not exist" Oct 30 22:07:49.401148 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.401092 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mfc\" (UniqueName: \"kubernetes.io/projected/d3222f4f-7d12-4ba8-943b-7bedc2102107-kube-api-access-s6mfc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:07:49.492503 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.492400 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv"] Oct 30 22:07:49.501525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:49.501466 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-97895160-675cbbd785-jzvkv"] Oct 30 22:07:51.467713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:51.467678 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3222f4f-7d12-4ba8-943b-7bedc2102107" path="/var/lib/kubelet/pods/d3222f4f-7d12-4ba8-943b-7bedc2102107/volumes" Oct 30 22:07:53.051349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:53.051314 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb"] Oct 30 22:07:53.101748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:53.101719 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7de04a9-mfqgb"] Oct 30 22:07:53.467209 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:07:53.467130 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53945f04-2719-4de3-872e-b06d21ee17ad" path="/var/lib/kubelet/pods/53945f04-2719-4de3-872e-b06d21ee17ad/volumes" Oct 30 22:07:58.550757 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:07:58.550719 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:08:00.153398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:00.153358 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2"] Oct 30 22:08:00.208282 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:00.208243 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d71f464-khfh2"] Oct 30 22:08:01.470086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:01.470048 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af6cbe0-852a-492f-a2c9-59213fb8e80f" path="/var/lib/kubelet/pods/1af6cbe0-852a-492f-a2c9-59213fb8e80f/volumes" Oct 30 22:08:08.361386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:08.361349 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x"] Oct 30 22:08:08.366707 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:08.366681 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-64086213-hqc6x"] Oct 30 22:08:08.383821 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:08.383799 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:08:09.477167 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:09.477134 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4b7561-a31f-4b13-a6c2-34948b57c0ea" path="/var/lib/kubelet/pods/7d4b7561-a31f-4b13-a6c2-34948b57c0ea/volumes" Oct 30 22:08:12.540762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:12.540716 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:12.766424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:12.766385 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:15.142657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:15.142620 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc"] Oct 30 22:08:15.190005 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:15.189969 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97895160-wfptc"] Oct 30 22:08:15.469812 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:15.469726 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" path="/var/lib/kubelet/pods/e250f7aa-1fb1-42a6-88a4-b6ee7953efe4/volumes" Oct 30 22:08:18.068940 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.068909 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx"] Oct 30 22:08:18.069624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069607 4933 cpu_manager.go:414] "containerMap: removing container" podUID="d3222f4f-7d12-4ba8-943b-7bedc2102107" containerName="payload-processing" Oct 30 22:08:18.069624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069623 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3222f4f-7d12-4ba8-943b-7bedc2102107" containerName="payload-processing" Oct 30 22:08:18.069765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069633 4933 cpu_manager.go:414] "containerMap: removing container" podUID="eaafe556-aa91-411a-bc1b-2baac76e8b40" containerName="payload-processing" Oct 30 22:08:18.069765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069638 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaafe556-aa91-411a-bc1b-2baac76e8b40" containerName="payload-processing" Oct 30 22:08:18.069765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069656 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" containerName="revoke-keys" Oct 30 22:08:18.069765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069661 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" containerName="revoke-keys" Oct 30 22:08:18.069765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069670 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ef410b18-912a-4d5f-a511-5115972a418b" containerName="payload-pre-processing" Oct 30 22:08:18.069765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069674 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef410b18-912a-4d5f-a511-5115972a418b" containerName="payload-pre-processing" Oct 30 22:08:18.070086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069791 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3222f4f-7d12-4ba8-943b-7bedc2102107" containerName="payload-processing" Oct 30 22:08:18.070086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069799 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaafe556-aa91-411a-bc1b-2baac76e8b40" containerName="payload-processing" Oct 30 22:08:18.070086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069809 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e250f7aa-1fb1-42a6-88a4-b6ee7953efe4" containerName="revoke-keys" Oct 30 22:08:18.070086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.069816 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef410b18-912a-4d5f-a511-5115972a418b" containerName="payload-pre-processing" Oct 30 22:08:18.075153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.075130 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.095852 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.095827 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hg4\" (UniqueName: \"kubernetes.io/projected/dec1458e-b90b-4db9-8351-f0dbe457a113-kube-api-access-74hg4\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.096049 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.096029 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.096092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.096073 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.132607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.132578 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx"] Oct 30 22:08:18.175790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.175762 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:08:18.175790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.175782 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:08:18.179918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.179894 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:08:18.179918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.179897 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:08:18.197729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.197708 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hg4\" (UniqueName: \"kubernetes.io/projected/dec1458e-b90b-4db9-8351-f0dbe457a113-kube-api-access-74hg4\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.197859 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.197848 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.197905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.197874 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.198477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.198459 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.200506 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.200473 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.256817 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.256784 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hg4\" (UniqueName: \"kubernetes.io/projected/dec1458e-b90b-4db9-8351-f0dbe457a113-kube-api-access-74hg4\") pod \"maas-api-revoke-sub-9f4ad945-9njnx\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.395227 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.395157 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:18.785296 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:08:18.785266 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec1458e_b90b_4db9_8351_f0dbe457a113.slice/crio-0ee28809d94ee003b0196b648c7a099d21ab68d17ed70c5744db1a7c70395235 WatchSource:0}: Error finding container 0ee28809d94ee003b0196b648c7a099d21ab68d17ed70c5744db1a7c70395235: Status 404 returned error can't find the container with id 0ee28809d94ee003b0196b648c7a099d21ab68d17ed70c5744db1a7c70395235 Oct 30 22:08:18.788472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.788452 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx"] Oct 30 22:08:18.870430 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:18.870400 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" event={"ID":"dec1458e-b90b-4db9-8351-f0dbe457a113","Type":"ContainerStarted","Data":"0ee28809d94ee003b0196b648c7a099d21ab68d17ed70c5744db1a7c70395235"} Oct 30 22:08:19.884415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:19.884373 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" event={"ID":"dec1458e-b90b-4db9-8351-f0dbe457a113","Type":"ContainerStarted","Data":"cf6518836ff86f5abfd5c969d840049f0854466a43e6622f4d0ec90e04d2bedd"} Oct 30 22:08:20.006051 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:20.006005 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" podStartSLOduration=2.005990813 podStartE2EDuration="2.005990813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:20.002917797 +0000 UTC m=+1548.684067720" watchObservedRunningTime="2026-10-30 22:08:20.005990813 +0000 UTC m=+1548.687140705" Oct 30 22:08:20.896540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:20.896505 4933 generic.go:356] "Generic (PLEG): container finished" podID="dec1458e-b90b-4db9-8351-f0dbe457a113" containerID="cf6518836ff86f5abfd5c969d840049f0854466a43e6622f4d0ec90e04d2bedd" exitCode=0 Oct 30 22:08:20.896935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:20.896561 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" event={"ID":"dec1458e-b90b-4db9-8351-f0dbe457a113","Type":"ContainerDied","Data":"cf6518836ff86f5abfd5c969d840049f0854466a43e6622f4d0ec90e04d2bedd"} Oct 30 22:08:22.079761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.079739 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:22.092495 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.092466 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq"] Oct 30 22:08:22.143368 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.143337 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014b72ba-mnbkq"] Oct 30 22:08:22.194965 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.194887 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-openshift-ca\") pod \"dec1458e-b90b-4db9-8351-f0dbe457a113\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " Oct 30 22:08:22.195116 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.195055 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-xks-ca\") pod \"dec1458e-b90b-4db9-8351-f0dbe457a113\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " Oct 30 22:08:22.195230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.195209 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec1458e-b90b-4db9-8351-f0dbe457a113-kube-api-access-74hg4\" (UniqueName: \"kubernetes.io/projected/dec1458e-b90b-4db9-8351-f0dbe457a113-kube-api-access-74hg4\") pod \"dec1458e-b90b-4db9-8351-f0dbe457a113\" (UID: \"dec1458e-b90b-4db9-8351-f0dbe457a113\") " Oct 30 22:08:22.195567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.195408 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-openshift-ca" pod "dec1458e-b90b-4db9-8351-f0dbe457a113" (UID: "dec1458e-b90b-4db9-8351-f0dbe457a113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:22.196234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.196189 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:22.197428 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.197405 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-xks-ca" pod "dec1458e-b90b-4db9-8351-f0dbe457a113" (UID: "dec1458e-b90b-4db9-8351-f0dbe457a113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:22.197826 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.197806 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec1458e-b90b-4db9-8351-f0dbe457a113-kube-api-access-74hg4" pod "dec1458e-b90b-4db9-8351-f0dbe457a113" (UID: "dec1458e-b90b-4db9-8351-f0dbe457a113"). InnerVolumeSpecName "kube-api-access-74hg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:22.298026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.297998 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74hg4\" (UniqueName: \"kubernetes.io/projected/dec1458e-b90b-4db9-8351-f0dbe457a113-kube-api-access-74hg4\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:22.298026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.298026 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec1458e-b90b-4db9-8351-f0dbe457a113-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:08:22.915765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.915737 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" Oct 30 22:08:22.915960 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.915780 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx" event={"ID":"dec1458e-b90b-4db9-8351-f0dbe457a113","Type":"ContainerDied","Data":"0ee28809d94ee003b0196b648c7a099d21ab68d17ed70c5744db1a7c70395235"} Oct 30 22:08:22.915960 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:22.915831 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee28809d94ee003b0196b648c7a099d21ab68d17ed70c5744db1a7c70395235" Oct 30 22:08:23.467550 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:23.467514 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a" path="/var/lib/kubelet/pods/5c3452cf-6fb9-4af1-9e9a-99d7dbac3a9a/volumes" Oct 30 22:08:28.547105 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:08:28.547069 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:08:33.080373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:33.080341 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d"] Oct 30 22:08:33.128843 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:33.128814 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ce908ea-glg6d"] Oct 30 22:08:33.467264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:33.467186 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104c12f7-935c-41e5-b8e2-8844f45a4ace" path="/var/lib/kubelet/pods/104c12f7-935c-41e5-b8e2-8844f45a4ace/volumes" Oct 30 22:08:39.715862 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:39.715830 4933 scope.go:122] "RemoveContainer" containerID="8d74e64084769a122c1a564d09846ae3a4b7cf80489dd7fac399b792fef1fd09" Oct 30 22:08:39.727522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:39.727480 4933 scope.go:122] "RemoveContainer" containerID="d68505ee3bd51d406dbef601fffe7fb9c14332c38b671249695d359e389e6923" Oct 30 22:08:48.156071 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:48.155952 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn"] Oct 30 22:08:48.211752 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:48.211720 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25222a92-pn7xn"] Oct 30 22:08:49.470388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:49.470350 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7d5514-83c1-488f-813a-542407800d2a" path="/var/lib/kubelet/pods/3b7d5514-83c1-488f-813a-542407800d2a/volumes" Oct 30 22:08:55.143082 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:55.143053 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm"] Oct 30 22:08:55.196410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:55.196372 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9896b16-zp5dm"] Oct 30 22:08:55.493231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:08:55.493062 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb2dadf-3719-45ea-b6de-6e031de9e683" path="/var/lib/kubelet/pods/cdb2dadf-3719-45ea-b6de-6e031de9e683/volumes" Oct 30 22:08:58.636728 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:08:58.636690 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:09:03.069657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:03.069616 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5"] Oct 30 22:09:03.119289 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:03.119259 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e4d0522-nqzv5"] Oct 30 22:09:03.468694 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:03.468624 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2" path="/var/lib/kubelet/pods/5fc8d3e4-62b8-4d4b-9bda-c23b635f36a2/volumes" Oct 30 22:09:06.053952 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:06.053919 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8"] Oct 30 22:09:06.104118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:06.104087 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ebeaac-gxwp8"] Oct 30 22:09:07.468958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:07.468924 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7da15a-872a-4552-85dd-dd465812ec50" path="/var/lib/kubelet/pods/be7da15a-872a-4552-85dd-dd465812ec50/volumes" Oct 30 22:09:11.092538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:11.092502 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw"] Oct 30 22:09:11.141777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:11.141752 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-126845c2-rdprw"] Oct 30 22:09:11.477885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:11.477793 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1f86c0-be28-4668-8a9e-0e8d71a94c77" path="/var/lib/kubelet/pods/cc1f86c0-be28-4668-8a9e-0e8d71a94c77/volumes" Oct 30 22:09:22.053747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:22.053717 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz"] Oct 30 22:09:22.103792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:22.103761 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb459a08-tgllz"] Oct 30 22:09:23.050770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:23.050740 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7"] Oct 30 22:09:23.100706 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:23.100677 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61b16424-smwn7"] Oct 30 22:09:23.469541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:23.469440 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88eb1189-4cd8-4546-affb-042fa0f0073f" path="/var/lib/kubelet/pods/88eb1189-4cd8-4546-affb-042fa0f0073f/volumes" Oct 30 22:09:23.469899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:23.469884 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9115f3ae-c5af-4754-8e06-bf1be900e777" path="/var/lib/kubelet/pods/9115f3ae-c5af-4754-8e06-bf1be900e777/volumes" Oct 30 22:09:26.050947 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:26.050911 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql"] Oct 30 22:09:26.084310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:26.084280 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3219452e-cw7ql"] Oct 30 22:09:27.051003 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:27.050970 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n"] Oct 30 22:09:27.098376 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:27.098346 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb66d6-6pq4n"] Oct 30 22:09:27.469211 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:27.469120 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9b4a49-9f70-4b43-b538-8ef7654eb444" path="/var/lib/kubelet/pods/cf9b4a49-9f70-4b43-b538-8ef7654eb444/volumes" Oct 30 22:09:27.469705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:27.469688 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb08710c-0cab-423f-b093-4b22255fc6a7" path="/var/lib/kubelet/pods/fb08710c-0cab-423f-b093-4b22255fc6a7/volumes" Oct 30 22:09:28.051735 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:28.051701 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp"] Oct 30 22:09:28.099396 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:28.099362 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9fe2816-6vcrp"] Oct 30 22:09:28.593475 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:09:28.593436 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:09:28.771758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:28.771724 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:09:29.482582 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:29.482550 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3705c57b-5255-41c4-be8a-5add2186e611" path="/var/lib/kubelet/pods/3705c57b-5255-41c4-be8a-5add2186e611/volumes" Oct 30 22:09:34.961797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:34.961767 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5"] Oct 30 22:09:34.962471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:34.962458 4933 cpu_manager.go:414] "containerMap: removing container" podUID="dec1458e-b90b-4db9-8351-f0dbe457a113" containerName="revoke-keys" Oct 30 22:09:34.962537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:34.962473 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec1458e-b90b-4db9-8351-f0dbe457a113" containerName="revoke-keys" Oct 30 22:09:34.962624 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:34.962613 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec1458e-b90b-4db9-8351-f0dbe457a113" containerName="revoke-keys" Oct 30 22:09:34.968149 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:34.968131 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.091906 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.091873 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch"] Oct 30 22:09:35.144526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.113954 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-242ddaf0-openshift-default-dockercfg-ljxqg\"" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.147722 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-credential-socket\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.147785 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-envoy\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.147823 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386b34f2-8410-4621-9678-4c7a1df8270f-istiod-ca-cert\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.147928 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-istio-token\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.147979 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-data\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.148033 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhv2c\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-kube-api-access-jhv2c\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.148178 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386b34f2-8410-4621-9678-4c7a1df8270f-istio-podinfo\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.148283 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-certs\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.149145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.148323 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-socket\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.163584 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.163554 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9d3856f-v9sch"] Oct 30 22:09:35.209577 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.209549 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5"] Oct 30 22:09:35.250080 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.249998 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386b34f2-8410-4621-9678-4c7a1df8270f-istio-podinfo\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250104 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-certs\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250148 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-socket\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250215 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-credential-socket\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250251 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-envoy\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250284 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386b34f2-8410-4621-9678-4c7a1df8270f-istiod-ca-cert\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250414 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250378 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-istio-token\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250426 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-data\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250481 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhv2c\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-kube-api-access-jhv2c\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250601 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250528 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-socket\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.250642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.250595 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-certs\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.251103 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.251065 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386b34f2-8410-4621-9678-4c7a1df8270f-istiod-ca-cert\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.251103 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.251072 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-data\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.251252 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.251109 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-credential-socket\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.253411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.253382 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386b34f2-8410-4621-9678-4c7a1df8270f-istio-podinfo\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.253728 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.253707 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-envoy\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.317553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.309090 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-istio-token\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.317553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.309775 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhv2c\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-kube-api-access-jhv2c\") pod \"e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:35.470111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.470080 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ffaecb-87c8-4a59-96ba-5820d4490e51" path="/var/lib/kubelet/pods/11ffaecb-87c8-4a59-96ba-5820d4490e51/volumes" Oct 30 22:09:35.591947 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:35.591910 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:36.075975 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:09:36.075935 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386b34f2_8410_4621_9678_4c7a1df8270f.slice/crio-4cfca5a87521ff64d9dc26b04eb6c8edb8ef4256500cdc7a1e9c85d6a033db5b WatchSource:0}: Error finding container 4cfca5a87521ff64d9dc26b04eb6c8edb8ef4256500cdc7a1e9c85d6a033db5b: Status 404 returned error can't find the container with id 4cfca5a87521ff64d9dc26b04eb6c8edb8ef4256500cdc7a1e9c85d6a033db5b Oct 30 22:09:36.079002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:36.078929 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5"] Oct 30 22:09:36.079331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:36.079292 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:36.079408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:36.079374 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:36.079470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:36.079413 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:09:36.958457 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:36.958421 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" event={"ID":"386b34f2-8410-4621-9678-4c7a1df8270f","Type":"ContainerStarted","Data":"c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4"} Oct 30 22:09:36.958751 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:36.958480 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" event={"ID":"386b34f2-8410-4621-9678-4c7a1df8270f","Type":"ContainerStarted","Data":"4cfca5a87521ff64d9dc26b04eb6c8edb8ef4256500cdc7a1e9c85d6a033db5b"} Oct 30 22:09:37.133539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:37.133473 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podStartSLOduration=3.133459606 podStartE2EDuration="3.133459606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:37.128374628 +0000 UTC m=+1625.809524516" watchObservedRunningTime="2026-10-30 22:09:37.133459606 +0000 UTC m=+1625.814609498" Oct 30 22:09:37.592668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:37.592632 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:37.598115 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:37.598091 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:37.968640 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:37.968562 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:37.969632 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:37.969615 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:09:38.054943 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:38.054917 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r"] Oct 30 22:09:38.078591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:38.078564 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9"] Oct 30 22:09:38.085769 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:38.085746 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38924ce4-jkh2r"] Oct 30 22:09:38.091714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:38.091693 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27f4cb5b-74xf9"] Oct 30 22:09:39.468837 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:39.468806 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="537c5c98-0834-4f21-b50c-d275c805eec3" path="/var/lib/kubelet/pods/537c5c98-0834-4f21-b50c-d275c805eec3/volumes" Oct 30 22:09:39.469302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:39.469283 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60224e1-9943-4edb-bc33-132f7e58f663" path="/var/lib/kubelet/pods/e60224e1-9943-4edb-bc33-132f7e58f663/volumes" Oct 30 22:09:39.781140 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:39.781112 4933 scope.go:122] "RemoveContainer" containerID="74c1e0b472508bad52c67f35d11c9e18df662aa67b4f60b81dda19008293eab0" Oct 30 22:09:39.793009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:39.792993 4933 scope.go:122] "RemoveContainer" containerID="cc7b4656a2a0c3df484f25ea0977ef754dae61e582dcfc48d4818bd3ca36a569" Oct 30 22:09:39.803203 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:39.803186 4933 scope.go:122] "RemoveContainer" containerID="15f01a0e90c8a053db00d800279d8d981eb378d68b238a09a8f287015bbc52fa" Oct 30 22:09:42.054646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:42.054612 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks"] Oct 30 22:09:42.104424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:42.104397 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0d838f-lqkks"] Oct 30 22:09:43.054981 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:43.054945 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw"] Oct 30 22:09:43.102531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:43.102501 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20f7e45-2vjfw"] Oct 30 22:09:43.470537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:43.470424 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4" path="/var/lib/kubelet/pods/5033c5b3-bfc5-46e3-9b5d-ad722ab3a2d4/volumes" Oct 30 22:09:43.471088 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:43.471068 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edba9717-763b-4c13-81fe-ff9e0f0405be" path="/var/lib/kubelet/pods/edba9717-763b-4c13-81fe-ff9e0f0405be/volumes" Oct 30 22:09:49.050665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:49.050632 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg"] Oct 30 22:09:49.100268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:49.100239 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4136dc62-n9zdg"] Oct 30 22:09:49.468708 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:49.468621 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b356211-dcdc-4f92-b4a7-86c48e30eb15" path="/var/lib/kubelet/pods/6b356211-dcdc-4f92-b4a7-86c48e30eb15/volumes" Oct 30 22:09:53.557159 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.557119 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l"] Oct 30 22:09:53.570658 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.570630 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.594905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.594873 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l"] Oct 30 22:09:53.638733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.638704 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:09:53.638936 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.638827 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:53.638936 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.638885 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:53.638936 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.638927 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:53.671812 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.671770 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.672011 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.671911 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.672173 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.672156 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tnn7\" (UniqueName: \"kubernetes.io/projected/05ee931f-7e95-4e24-82a2-04c072f09d87-kube-api-access-8tnn7\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.773743 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.773714 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tnn7\" (UniqueName: \"kubernetes.io/projected/05ee931f-7e95-4e24-82a2-04c072f09d87-kube-api-access-8tnn7\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.773911 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.773795 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.773911 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.773856 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.774435 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.774414 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.776344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.776310 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.831432 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.831374 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tnn7\" (UniqueName: \"kubernetes.io/projected/05ee931f-7e95-4e24-82a2-04c072f09d87-kube-api-access-8tnn7\") pod \"maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:53.895824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:53.895801 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:54.278031 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:09:54.278004 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ee931f_7e95_4e24_82a2_04c072f09d87.slice/crio-898c68299fe7261c02784b5097c11f6e0562fee5dc81ae213d5497a16fb637a1 WatchSource:0}: Error finding container 898c68299fe7261c02784b5097c11f6e0562fee5dc81ae213d5497a16fb637a1: Status 404 returned error can't find the container with id 898c68299fe7261c02784b5097c11f6e0562fee5dc81ae213d5497a16fb637a1 Oct 30 22:09:54.281474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:54.281425 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l"] Oct 30 22:09:55.138327 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:55.138298 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" event={"ID":"05ee931f-7e95-4e24-82a2-04c072f09d87","Type":"ContainerStarted","Data":"dd7d2b21e747a033035fdd23e2131ab61c0ee9331ef2b45ca4134560a5224a9f"} Oct 30 22:09:55.138806 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:55.138356 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" event={"ID":"05ee931f-7e95-4e24-82a2-04c072f09d87","Type":"ContainerStarted","Data":"898c68299fe7261c02784b5097c11f6e0562fee5dc81ae213d5497a16fb637a1"} Oct 30 22:09:56.152139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:56.152103 4933 generic.go:356] "Generic (PLEG): container finished" podID="05ee931f-7e95-4e24-82a2-04c072f09d87" containerID="dd7d2b21e747a033035fdd23e2131ab61c0ee9331ef2b45ca4134560a5224a9f" exitCode=0 Oct 30 22:09:56.152659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:56.152201 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" event={"ID":"05ee931f-7e95-4e24-82a2-04c072f09d87","Type":"ContainerDied","Data":"dd7d2b21e747a033035fdd23e2131ab61c0ee9331ef2b45ca4134560a5224a9f"} Oct 30 22:09:57.348681 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.348401 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:57.425447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.425384 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-xks-ca\") pod \"05ee931f-7e95-4e24-82a2-04c072f09d87\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " Oct 30 22:09:57.425644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.425630 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ee931f-7e95-4e24-82a2-04c072f09d87-kube-api-access-8tnn7\" (UniqueName: \"kubernetes.io/projected/05ee931f-7e95-4e24-82a2-04c072f09d87-kube-api-access-8tnn7\") pod \"05ee931f-7e95-4e24-82a2-04c072f09d87\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " Oct 30 22:09:57.425722 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.425711 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-openshift-ca\") pod \"05ee931f-7e95-4e24-82a2-04c072f09d87\" (UID: \"05ee931f-7e95-4e24-82a2-04c072f09d87\") " Oct 30 22:09:57.426153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.426119 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-openshift-ca" pod "05ee931f-7e95-4e24-82a2-04c072f09d87" (UID: "05ee931f-7e95-4e24-82a2-04c072f09d87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:57.426298 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.426283 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:57.427755 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.427726 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-xks-ca" pod "05ee931f-7e95-4e24-82a2-04c072f09d87" (UID: "05ee931f-7e95-4e24-82a2-04c072f09d87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:57.427844 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.427808 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ee931f-7e95-4e24-82a2-04c072f09d87-kube-api-access-8tnn7" pod "05ee931f-7e95-4e24-82a2-04c072f09d87" (UID: "05ee931f-7e95-4e24-82a2-04c072f09d87"). InnerVolumeSpecName "kube-api-access-8tnn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:57.528417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.528308 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tnn7\" (UniqueName: \"kubernetes.io/projected/05ee931f-7e95-4e24-82a2-04c072f09d87-kube-api-access-8tnn7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:57.528417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:57.528344 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ee931f-7e95-4e24-82a2-04c072f09d87-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:09:58.073734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:58.073690 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p"] Oct 30 22:09:58.126177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:58.126144 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3a35d24-pp75p"] Oct 30 22:09:58.176384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:58.176344 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" event={"ID":"05ee931f-7e95-4e24-82a2-04c072f09d87","Type":"ContainerDied","Data":"898c68299fe7261c02784b5097c11f6e0562fee5dc81ae213d5497a16fb637a1"} Oct 30 22:09:58.176384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:58.176382 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l" Oct 30 22:09:58.176646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:58.176406 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="898c68299fe7261c02784b5097c11f6e0562fee5dc81ae213d5497a16fb637a1" Oct 30 22:09:58.546791 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:09:58.546753 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:09:59.114710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:59.114682 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6"] Oct 30 22:09:59.114894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:59.114872 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" containerID="cri-o://cd12e4d0d6ab5d1c37a27109ed9cedc7083e3ad4850be3aa3b9a273854607acc" gracePeriod=30 Oct 30 22:09:59.480168 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:09:59.479132 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edbffec-25c8-483f-aa6a-259737a28b10" path="/var/lib/kubelet/pods/9edbffec-25c8-483f-aa6a-259737a28b10/volumes" Oct 30 22:10:02.054621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:02.054576 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz"] Oct 30 22:10:02.103983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:02.103954 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0cda51d-sclgz"] Oct 30 22:10:03.051933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:03.051892 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp"] Oct 30 22:10:03.102863 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:03.102828 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6648aab-b4lzp"] Oct 30 22:10:03.123620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:03.123588 4933 patch_prober.go:29] interesting pod/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:10:03.123768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:03.123643 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:10:03.475542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:03.475441 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294ec9c2-60c0-4b7d-a6af-2cfdc7478467" path="/var/lib/kubelet/pods/294ec9c2-60c0-4b7d-a6af-2cfdc7478467/volumes" Oct 30 22:10:03.475942 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:03.475927 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ded627-4bad-4d13-96a2-fef1fb1b7452" path="/var/lib/kubelet/pods/79ded627-4bad-4d13-96a2-fef1fb1b7452/volumes" Oct 30 22:10:04.236321 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.236296 4933 generic.go:356] "Generic (PLEG): container finished" podID="1132ebc6-e825-4a81-9048-0119ce05028c" containerID="cd12e4d0d6ab5d1c37a27109ed9cedc7083e3ad4850be3aa3b9a273854607acc" exitCode=0 Oct 30 22:10:04.236743 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.236369 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" event={"ID":"1132ebc6-e825-4a81-9048-0119ce05028c","Type":"ContainerDied","Data":"cd12e4d0d6ab5d1c37a27109ed9cedc7083e3ad4850be3aa3b9a273854607acc"} Oct 30 22:10:04.407661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.407639 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:10:04.531782 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.531743 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1132ebc6-e825-4a81-9048-0119ce05028c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132ebc6-e825-4a81-9048-0119ce05028c-istiod-ca-cert\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.532003 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.531865 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-kube-api-access-fnm9h\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-kube-api-access-fnm9h\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.532003 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.531925 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-credential-socket\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.532196 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.532168 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1132ebc6-e825-4a81-9048-0119ce05028c-istiod-ca-cert" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:04.532313 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.532265 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-credential-socket" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:04.533710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.532042 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-certs\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.533845 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.533749 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-envoy\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.533845 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.533757 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-certs" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:04.534464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.534193 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1132ebc6-e825-4a81-9048-0119ce05028c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132ebc6-e825-4a81-9048-0119ce05028c-istio-podinfo\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.534714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.534698 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-data\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.534964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.534943 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-istio-token\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-istio-token\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.535646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.535627 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-socket\") pod \"1132ebc6-e825-4a81-9048-0119ce05028c\" (UID: \"1132ebc6-e825-4a81-9048-0119ce05028c\") " Oct 30 22:10:04.535930 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.535887 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-socket" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:04.535930 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.535665 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-kube-api-access-fnm9h" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "kube-api-access-fnm9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:04.536637 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.536611 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1132ebc6-e825-4a81-9048-0119ce05028c-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.536745 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.536643 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnm9h\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-kube-api-access-fnm9h\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.536745 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.536658 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.536745 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.536676 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.536745 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.536690 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.537947 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.537925 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-data" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:04.546900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.544749 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-envoy" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:04.547466 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.547441 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-istio-token" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:04.548899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.548863 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1132ebc6-e825-4a81-9048-0119ce05028c-istio-podinfo" pod "1132ebc6-e825-4a81-9048-0119ce05028c" (UID: "1132ebc6-e825-4a81-9048-0119ce05028c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:04.638819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.638675 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.638819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.638699 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1132ebc6-e825-4a81-9048-0119ce05028c-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.638819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.638713 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1132ebc6-e825-4a81-9048-0119ce05028c-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:04.638819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:04.638725 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1132ebc6-e825-4a81-9048-0119ce05028c-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:05.248162 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:05.248132 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" Oct 30 22:10:05.248788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:05.248164 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6" event={"ID":"1132ebc6-e825-4a81-9048-0119ce05028c","Type":"ContainerDied","Data":"23aed8148af31d66ba02743a5be183235b8602dab7e2f3184c710761e86e21bc"} Oct 30 22:10:05.248788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:05.248230 4933 scope.go:122] "RemoveContainer" containerID="cd12e4d0d6ab5d1c37a27109ed9cedc7083e3ad4850be3aa3b9a273854607acc" Oct 30 22:10:05.649558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:05.648965 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6"] Oct 30 22:10:05.649558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:05.649018 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-4whr6"] Oct 30 22:10:06.121605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:06.121572 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz"] Oct 30 22:10:06.134051 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:06.126467 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-090d71e8-lv9dz"] Oct 30 22:10:07.130450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:07.130410 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n"] Oct 30 22:10:07.178036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:07.178004 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d12c7586-hxh8n"] Oct 30 22:10:07.469057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:07.468985 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" path="/var/lib/kubelet/pods/1132ebc6-e825-4a81-9048-0119ce05028c/volumes" Oct 30 22:10:07.469442 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:07.469428 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f60c16-dd5b-40e8-9c68-fa611500376f" path="/var/lib/kubelet/pods/23f60c16-dd5b-40e8-9c68-fa611500376f/volumes" Oct 30 22:10:07.469910 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:07.469897 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e06ae8-93a9-43f6-a9cd-8de09b85fb2f" path="/var/lib/kubelet/pods/79e06ae8-93a9-43f6-a9cd-8de09b85fb2f/volumes" Oct 30 22:10:11.054519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:11.054413 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn"] Oct 30 22:10:11.101944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:11.101914 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc8b8bfe-9z7zn"] Oct 30 22:10:11.474166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:11.474096 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d483ee-1f1d-4a19-bbdf-7ef619280ec4" path="/var/lib/kubelet/pods/04d483ee-1f1d-4a19-bbdf-7ef619280ec4/volumes" Oct 30 22:10:12.350300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.350271 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc"] Oct 30 22:10:12.351124 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.351101 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" Oct 30 22:10:12.351232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.351125 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" Oct 30 22:10:12.351232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.351156 4933 cpu_manager.go:414] "containerMap: removing container" podUID="05ee931f-7e95-4e24-82a2-04c072f09d87" containerName="revoke-keys" Oct 30 22:10:12.351232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.351163 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ee931f-7e95-4e24-82a2-04c072f09d87" containerName="revoke-keys" Oct 30 22:10:12.351385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.351377 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1132ebc6-e825-4a81-9048-0119ce05028c" containerName="istio-proxy" Oct 30 22:10:12.351438 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.351393 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ee931f-7e95-4e24-82a2-04c072f09d87" containerName="revoke-keys" Oct 30 22:10:12.356655 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.356635 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.411813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.411787 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc"] Oct 30 22:10:12.412456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.412431 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:10:12.412607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.412442 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:10:12.412607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.412576 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:10:12.436666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.436638 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.436804 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.436742 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8rq\" (UniqueName: \"kubernetes.io/projected/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-kube-api-access-2t8rq\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.436909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.436892 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.456604 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.456579 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:10:12.538440 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.538414 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.538637 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.538618 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.538740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.538724 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8rq\" (UniqueName: \"kubernetes.io/projected/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-kube-api-access-2t8rq\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.539387 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.539362 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.541248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.541223 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.613252 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.613167 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8rq\" (UniqueName: \"kubernetes.io/projected/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-kube-api-access-2t8rq\") pod \"maas-api-revoke-keys-e2e-derive-497924bf-lk2dc\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:12.671949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:12.671919 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:13.073121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:13.073089 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc"] Oct 30 22:10:13.320273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:13.320201 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" event={"ID":"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed","Type":"ContainerStarted","Data":"9297d009ecd519b219eea3dd3b26d144af5badddd6a3fbd15c9134b30d5e69d3"} Oct 30 22:10:14.329651 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:14.329620 4933 generic.go:356] "Generic (PLEG): container finished" podID="1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" containerID="245e6e84da6736cd22110c209b46b0847120bd92f2325029bcf3bb4e9aecdf3f" exitCode=0 Oct 30 22:10:14.330036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:14.329721 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" event={"ID":"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed","Type":"ContainerDied","Data":"245e6e84da6736cd22110c209b46b0847120bd92f2325029bcf3bb4e9aecdf3f"} Oct 30 22:10:15.684832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.679101 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:15.712449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.712420 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-kube-api-access-2t8rq\" (UniqueName: \"kubernetes.io/projected/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-kube-api-access-2t8rq\") pod \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " Oct 30 22:10:15.712675 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.712467 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-xks-ca\") pod \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " Oct 30 22:10:15.712675 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.712515 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-openshift-ca\") pod \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\" (UID: \"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed\") " Oct 30 22:10:15.712956 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.712932 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-openshift-ca" pod "1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" (UID: "1b2b2ae6-303b-4cc1-ae81-c260f15f13ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:15.713050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.713037 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.715351 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.715327 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-xks-ca" pod "1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" (UID: "1b2b2ae6-303b-4cc1-ae81-c260f15f13ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:10:15.715514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.715381 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-kube-api-access-2t8rq" pod "1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" (UID: "1b2b2ae6-303b-4cc1-ae81-c260f15f13ed"). InnerVolumeSpecName "kube-api-access-2t8rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:15.819552 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.816763 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t8rq\" (UniqueName: \"kubernetes.io/projected/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-kube-api-access-2t8rq\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:15.819552 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:15.816800 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:10:16.364036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:16.364007 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" Oct 30 22:10:16.364233 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:16.364040 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc" event={"ID":"1b2b2ae6-303b-4cc1-ae81-c260f15f13ed","Type":"ContainerDied","Data":"9297d009ecd519b219eea3dd3b26d144af5badddd6a3fbd15c9134b30d5e69d3"} Oct 30 22:10:16.364233 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:16.364090 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9297d009ecd519b219eea3dd3b26d144af5badddd6a3fbd15c9134b30d5e69d3" Oct 30 22:10:17.054559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:17.054523 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf"] Oct 30 22:10:17.086554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:17.086515 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7"] Oct 30 22:10:17.106754 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:17.106722 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f771a7e9-b58nf"] Oct 30 22:10:17.110745 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:17.110722 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5cc6b1-fxjn7"] Oct 30 22:10:17.493592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:17.490131 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cac580-21ff-424a-a265-cbde8e89c5d9" path="/var/lib/kubelet/pods/25cac580-21ff-424a-a265-cbde8e89c5d9/volumes" Oct 30 22:10:17.493592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:17.490834 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9df465-8f8c-4b67-a58b-69f43046c734" path="/var/lib/kubelet/pods/4e9df465-8f8c-4b67-a58b-69f43046c734/volumes" Oct 30 22:10:20.868171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:20.868130 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:10:21.052761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:21.052724 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf"] Oct 30 22:10:21.059680 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:21.059649 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d60d8d1-5mktf"] Oct 30 22:10:21.564295 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:21.564268 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15b39c0-2619-4289-b628-dc2e1e25c341" path="/var/lib/kubelet/pods/c15b39c0-2619-4289-b628-dc2e1e25c341/volumes" Oct 30 22:10:28.559134 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:10:28.559095 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:10:39.850483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.850451 4933 scope.go:122] "RemoveContainer" containerID="03cf126c80184633ccf91e0caeb6438c67e64d7a0144ef4388ba4d901c0c961c" Oct 30 22:10:39.864430 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.864409 4933 scope.go:122] "RemoveContainer" containerID="48d94768aa2d3851d4de52a465f60febe9185a89f4f0eeaf4aa71932c2b30f74" Oct 30 22:10:39.874771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.874748 4933 scope.go:122] "RemoveContainer" containerID="df94b3eb0d01997180fb3e0acab63c84839f92d04fe9d693825052c909d6c1cb" Oct 30 22:10:39.884420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.884402 4933 scope.go:122] "RemoveContainer" containerID="1f085afa259f76b70ef613491ebfb82a901335cb5eb06cc46ebc273ada0126b1" Oct 30 22:10:39.894033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.894013 4933 scope.go:122] "RemoveContainer" containerID="c1e4af932a6efb4cf96a494b62114245ad95257ed16c64e24795c9d96dc74d69" Oct 30 22:10:39.903522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.903500 4933 scope.go:122] "RemoveContainer" containerID="32dfb4ceaaf43d5cad7cf3dceafc41cd99f8fa1fc1696444d2268a061fa41c02" Oct 30 22:10:39.913057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.913043 4933 scope.go:122] "RemoveContainer" containerID="8940729a368601be36ee9646039fe79b0f6090edccb11562432ffa8d03d09a5e" Oct 30 22:10:39.922095 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.922057 4933 scope.go:122] "RemoveContainer" containerID="9ca6d219fa1c6949f1a400dcb0ecb4056f26cc7f147b36bd8a454faab9f91d1c" Oct 30 22:10:39.931036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.931015 4933 scope.go:122] "RemoveContainer" containerID="b690102398fefd437f472162939e582d26c95ddc3ebafebeb2cc62eb489cf55c" Oct 30 22:10:39.943416 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.943398 4933 scope.go:122] "RemoveContainer" containerID="938e6b90ab063d522f1b864a7d24d1d6706adba1ebf68e39d64d652401f4b261" Oct 30 22:10:39.954998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.954981 4933 scope.go:122] "RemoveContainer" containerID="15fb087cc3373bd378c1251ea3755f8190f23111546cd53cd7ac0e80c307bedc" Oct 30 22:10:39.964883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.964864 4933 scope.go:122] "RemoveContainer" containerID="b37c09a19e5c0a81566ff9b11ab6ea5b6be1777481ff43cbf9b04b6f5677edc3" Oct 30 22:10:39.976715 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.976695 4933 scope.go:122] "RemoveContainer" containerID="db090e405d4144362fae730c97f702a81f383302ba4cc7d70c535ccca43c3863" Oct 30 22:10:39.986279 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.986246 4933 scope.go:122] "RemoveContainer" containerID="e13779ed62f8304bdcbf548989a542bea3f243ad07f27da340ef40478a83f797" Oct 30 22:10:39.995431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:39.995414 4933 scope.go:122] "RemoveContainer" containerID="b0e1eab495d8af99816012f5d8469a3be4094822f9ec51f2834fdc1401d31f93" Oct 30 22:10:46.050788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:46.050753 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf"] Oct 30 22:10:46.080567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:46.080540 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a1c00c0-g7bxf"] Oct 30 22:10:47.467182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:47.467146 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf481b7-c320-492e-8514-79162382584c" path="/var/lib/kubelet/pods/cdf481b7-c320-492e-8514-79162382584c/volumes" Oct 30 22:10:52.038024 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:52.037985 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l"] Oct 30 22:10:52.108364 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:52.108321 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-90f9e9d3-gkv4l"] Oct 30 22:10:53.474554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:53.473985 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ee931f-7e95-4e24-82a2-04c072f09d87" path="/var/lib/kubelet/pods/05ee931f-7e95-4e24-82a2-04c072f09d87/volumes" Oct 30 22:10:58.547888 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:10:58.547837 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:10:58.866623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:58.866539 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff"] Oct 30 22:10:58.866832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:10:58.866808 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" containerID="cri-o://14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1" gracePeriod=30 Oct 30 22:11:04.099095 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.099058 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:11:04.099740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.099706 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 22:11:04.519927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.519894 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:11:04.716925 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.716843 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-token\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-token\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717218 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istiod-ca-cert\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717502 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717470 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-certs\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717569 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-socket\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717613 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-envoy\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717712 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-kube-api-access-nqdnq\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717741 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-podinfo\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717805 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-data\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.717889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717849 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-credential-socket\") pod \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\" (UID: \"488cb661-6f7f-4ba3-b7d2-826dc69fcd94\") " Oct 30 22:11:04.718056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.717736 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istiod-ca-cert" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:04.718134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.718024 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-socket" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.718313 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.718291 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-credential-socket" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.718757 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.718727 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-data" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.718860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.718808 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-certs" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.718961 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.718940 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.719060 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.719045 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.719159 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.719147 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.722631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.722594 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-token" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.722631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.722600 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-envoy" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:04.724405 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.724350 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-podinfo" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:04.724965 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.724929 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-kube-api-access-nqdnq" pod "488cb661-6f7f-4ba3-b7d2-826dc69fcd94" (UID: "488cb661-6f7f-4ba3-b7d2-826dc69fcd94"). InnerVolumeSpecName "kube-api-access-nqdnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:04.841515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.839664 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.841515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.839699 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.841515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.839713 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.841515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.839758 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqdnq\" (UniqueName: \"kubernetes.io/projected/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-kube-api-access-nqdnq\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.841515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.839775 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.841515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.839789 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/488cb661-6f7f-4ba3-b7d2-826dc69fcd94-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:04.934511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.933854 4933 generic.go:356] "Generic (PLEG): container finished" podID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerID="14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1" exitCode=0 Oct 30 22:11:04.934511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.933940 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" Oct 30 22:11:04.934511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.934144 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" event={"ID":"488cb661-6f7f-4ba3-b7d2-826dc69fcd94","Type":"ContainerDied","Data":"14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1"} Oct 30 22:11:04.934511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.934261 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff" event={"ID":"488cb661-6f7f-4ba3-b7d2-826dc69fcd94","Type":"ContainerDied","Data":"2d4738b0f441424af7c1c27e3738e90a9362f7b4e2c765ee14f1a534c4abcdeb"} Oct 30 22:11:04.934511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.934294 4933 scope.go:122] "RemoveContainer" containerID="14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1" Oct 30 22:11:04.973272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.973245 4933 scope.go:122] "RemoveContainer" containerID="14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1" Oct 30 22:11:04.975538 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:11:04.975371 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1\": container with ID starting with 14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1 not found: ID does not exist" containerID="14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1" Oct 30 22:11:04.975538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:04.975411 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1"} err="failed to get container status \"14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1\": rpc error: code = NotFound desc = could not find container \"14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1\": container with ID starting with 14ed7af1b75a3e0cb7d7f51187c132384817d125dce50abb32dcbfa6572bf0f1 not found: ID does not exist" Oct 30 22:11:05.062316 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:05.062277 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc"] Oct 30 22:11:05.068904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:05.068872 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-497924bf-lk2dc"] Oct 30 22:11:05.103747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:05.103718 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff"] Oct 30 22:11:05.108426 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:05.108298 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-fpvff"] Oct 30 22:11:05.478999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:05.478964 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" path="/var/lib/kubelet/pods/1b2b2ae6-303b-4cc1-ae81-c260f15f13ed/volumes" Oct 30 22:11:05.480070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:05.480034 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" path="/var/lib/kubelet/pods/488cb661-6f7f-4ba3-b7d2-826dc69fcd94/volumes" Oct 30 22:11:13.111752 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:13.111724 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:11:13.374627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:13.364860 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td"] Oct 30 22:11:13.374627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:13.365116 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" containerID="cri-o://2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c" gracePeriod=30 Oct 30 22:11:13.412516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:13.403248 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs"] Oct 30 22:11:13.426513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:13.416579 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" containerID="cri-o://55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34" gracePeriod=30 Oct 30 22:11:18.816830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.816792 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:11:18.879909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.879885 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:11:18.987199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986180 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-socket\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.987199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986235 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-envoy\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.987199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986263 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-data\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.987199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986323 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-envoy\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.987199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986429 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-credential-socket\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.987646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986476 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-kube-api-access-854xj\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-kube-api-access-854xj\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.987777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986558 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-socket" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.987777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.986836 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-data" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.987777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.987020 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-credential-socket" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.989277 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.987997 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-kube-api-access-bgj62\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-kube-api-access-bgj62\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.989903 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.989475 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-podinfo\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.990280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.990239 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-data" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.990463 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.990415 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-data\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.990869 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.990618 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-credential-socket\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.990869 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.990843 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-credential-socket" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.991399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991099 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-podinfo\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991547 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-socket\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991597 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istiod-ca-cert\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991644 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-certs\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991678 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istiod-ca-cert\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991709 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-certs\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991758 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-token\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-token\") pod \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\" (UID: \"c883ffae-9b9a-4622-9199-9aebffa4a5e6\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.991851 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-token\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-token\") pod \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\" (UID: \"fa7443bc-8680-45f0-b375-2d1c7a8d6dac\") " Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.992676 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.992875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.992698 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.995564 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.995536 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.995666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.995569 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.995666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.995586 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:18.995666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.993953 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-certs" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.995666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.994152 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-socket" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.995666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.994410 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istiod-ca-cert" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:18.995666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.994656 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-certs" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:18.995666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:18.994898 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istiod-ca-cert" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.013886 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-podinfo" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.016579 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-envoy" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.016598 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-token" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.016638 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-kube-api-access-bgj62" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "kube-api-access-bgj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.016713 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-podinfo" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.016736 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-envoy" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.016811 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-token" pod "c883ffae-9b9a-4622-9199-9aebffa4a5e6" (UID: "c883ffae-9b9a-4622-9199-9aebffa4a5e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:19.017281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.016866 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-kube-api-access-854xj" pod "fa7443bc-8680-45f0-b375-2d1c7a8d6dac" (UID: "fa7443bc-8680-45f0-b375-2d1c7a8d6dac"). InnerVolumeSpecName "kube-api-access-854xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:19.097404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097372 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097402 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097412 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097422 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854xj\" (UniqueName: \"kubernetes.io/projected/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-kube-api-access-854xj\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097432 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgj62\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-kube-api-access-bgj62\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097441 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097449 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097458 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097467 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097477 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c883ffae-9b9a-4622-9199-9aebffa4a5e6-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097503 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097516 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7443bc-8680-45f0-b375-2d1c7a8d6dac-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.097665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.097527 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c883ffae-9b9a-4622-9199-9aebffa4a5e6-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:11:19.127333 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.127302 4933 generic.go:356] "Generic (PLEG): container finished" podID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerID="55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34" exitCode=0 Oct 30 22:11:19.127576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.127356 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" Oct 30 22:11:19.127576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.127520 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" event={"ID":"fa7443bc-8680-45f0-b375-2d1c7a8d6dac","Type":"ContainerDied","Data":"55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34"} Oct 30 22:11:19.127576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.127570 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs" event={"ID":"fa7443bc-8680-45f0-b375-2d1c7a8d6dac","Type":"ContainerDied","Data":"0001f93fa050775210d0c604fffab7004983530c4ad2d7fd484a57bc6588d1be"} Oct 30 22:11:19.127780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.127599 4933 scope.go:122] "RemoveContainer" containerID="55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34" Oct 30 22:11:19.167563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.167183 4933 generic.go:356] "Generic (PLEG): container finished" podID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerID="2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c" exitCode=0 Oct 30 22:11:19.167563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.167293 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" event={"ID":"c883ffae-9b9a-4622-9199-9aebffa4a5e6","Type":"ContainerDied","Data":"2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c"} Oct 30 22:11:19.167563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.167334 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" event={"ID":"c883ffae-9b9a-4622-9199-9aebffa4a5e6","Type":"ContainerDied","Data":"2c569cb956fc7548224c8e8f8719c53da2fa9fde34464260264954e2f024dc22"} Oct 30 22:11:19.167563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.167390 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td" Oct 30 22:11:19.175087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.175066 4933 scope.go:122] "RemoveContainer" containerID="55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34" Oct 30 22:11:19.175628 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:11:19.175605 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34\": container with ID starting with 55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34 not found: ID does not exist" containerID="55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34" Oct 30 22:11:19.175791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.175768 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34"} err="failed to get container status \"55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34\": rpc error: code = NotFound desc = could not find container \"55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34\": container with ID starting with 55571d1ffaa5c2d18e44db4f84c841f602316a6475233383c87093b76a450b34 not found: ID does not exist" Oct 30 22:11:19.175882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.175872 4933 scope.go:122] "RemoveContainer" containerID="2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c" Oct 30 22:11:19.194174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.194022 4933 scope.go:122] "RemoveContainer" containerID="2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c" Oct 30 22:11:19.194820 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:11:19.194788 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c\": container with ID starting with 2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c not found: ID does not exist" containerID="2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c" Oct 30 22:11:19.194926 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.194832 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c"} err="failed to get container status \"2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c\": rpc error: code = NotFound desc = could not find container \"2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c\": container with ID starting with 2ddfa0b9619ffc5021c1da274f6c49637b9a164bdcd5efd5851a959de0907b1c not found: ID does not exist" Oct 30 22:11:19.366651 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.366612 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs"] Oct 30 22:11:19.370303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.370282 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-k67rs"] Oct 30 22:11:19.479853 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.479819 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" path="/var/lib/kubelet/pods/fa7443bc-8680-45f0-b375-2d1c7a8d6dac/volumes" Oct 30 22:11:19.480185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.480170 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td"] Oct 30 22:11:19.482896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:19.482880 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-xx7td"] Oct 30 22:11:21.510674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:21.510642 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" path="/var/lib/kubelet/pods/c883ffae-9b9a-4622-9199-9aebffa4a5e6/volumes" Oct 30 22:11:24.010646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:24.010605 4933 patch_prober.go:29] interesting pod/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:11:24.011073 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:24.010675 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:25.235974 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:25.235882 4933 generic.go:356] "Generic (PLEG): container finished" podID="386b34f2-8410-4621-9678-4c7a1df8270f" containerID="c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4" exitCode=0 Oct 30 22:11:25.235974 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:25.235957 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" event={"ID":"386b34f2-8410-4621-9678-4c7a1df8270f","Type":"ContainerDied","Data":"c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4"} Oct 30 22:11:25.236456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:25.236391 4933 scope.go:122] "RemoveContainer" containerID="c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4" Oct 30 22:11:25.238180 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:25.238146 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:25.238303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:25.238242 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:25.238373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:25.238293 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:11:26.256154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:26.255381 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" event={"ID":"386b34f2-8410-4621-9678-4c7a1df8270f","Type":"ContainerStarted","Data":"a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a"} Oct 30 22:11:26.592993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:26.592961 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:11:26.593174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:26.593110 4933 patch_prober.go:29] interesting pod/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:11:26.593238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:26.593182 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:11:27.593270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:27.593241 4933 patch_prober.go:29] interesting pod/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:11:27.593668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:27.593299 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:11:28.570659 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:11:28.570624 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:11:28.592924 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:28.592888 4933 patch_prober.go:29] interesting pod/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:11:28.593101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:28.592964 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:11:29.592599 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:29.592565 4933 patch_prober.go:29] interesting pod/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:11:29.593007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:29.592621 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:11:30.592750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:30.592719 4933 patch_prober.go:29] interesting pod/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 22:11:30.593220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:30.592776 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 22:11:31.671412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:31.671377 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:11:31.672424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:31.672391 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:11:31.672743 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:31.672720 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:11:40.117267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.117238 4933 scope.go:122] "RemoveContainer" containerID="b0890bd965f27f5ec9976aa7785d51e853c78e157d12f98728482239c1b34c32" Oct 30 22:11:40.128578 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.128556 4933 scope.go:122] "RemoveContainer" containerID="394b49615d2ea5bafc2d8d98f14af40cf41303601a461ab3ba9419fa965abc1a" Oct 30 22:11:40.138509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.138470 4933 scope.go:122] "RemoveContainer" containerID="c9d6665d56d3f4e29170d8ef3ef69f7dba3c9709d221ac6008d37c83c26d6b65" Oct 30 22:11:40.150085 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.150064 4933 scope.go:122] "RemoveContainer" containerID="87baebb315fe56a177e07803f7cb7b21885eda5f346d8fac438fea8d76016fa8" Oct 30 22:11:40.162256 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.162232 4933 scope.go:122] "RemoveContainer" containerID="cae0c3bb65d7f9f19bf8391d81e41e7ab4231364d38ae01d940cbf5937a8b53b" Oct 30 22:11:40.173410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.173392 4933 scope.go:122] "RemoveContainer" containerID="a2901e5fd1d1c58578ebe8f848401f7052741c2de51e576170da2a6c80dff1da" Oct 30 22:11:40.183820 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.183804 4933 scope.go:122] "RemoveContainer" containerID="e8a436ab77d8d1de24f4b467e79c8a6f0a72f721ee74c99e73eb929d360e13b3" Oct 30 22:11:40.192986 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.192965 4933 scope.go:122] "RemoveContainer" containerID="cc0c68f3fd3e3eabeea8090d64f8d5afd11b8e33525b574c64676129c5890e29" Oct 30 22:11:40.202688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.202674 4933 scope.go:122] "RemoveContainer" containerID="b33759cd8047e30d3b7e196728804a23e820b398d41a4d92cf0a2ddfebaa2058" Oct 30 22:11:40.215847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:11:40.215827 4933 scope.go:122] "RemoveContainer" containerID="7d71f3a6135ac10e495fe5a2bcef9e2d9517407838f5390ded4b3bbaf6e92a32" Oct 30 22:11:58.582011 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:11:58.581961 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:12:06.146883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.146846 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm"] Oct 30 22:12:06.147935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147910 4933 cpu_manager.go:414] "containerMap: removing container" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" Oct 30 22:12:06.147935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147933 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" Oct 30 22:12:06.148077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147950 4933 cpu_manager.go:414] "containerMap: removing container" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" Oct 30 22:12:06.148077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147955 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" Oct 30 22:12:06.148077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147972 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" containerName="revoke-keys" Oct 30 22:12:06.148077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147977 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" containerName="revoke-keys" Oct 30 22:12:06.148077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147986 4933 cpu_manager.go:414] "containerMap: removing container" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" Oct 30 22:12:06.148077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.147992 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" Oct 30 22:12:06.148264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.148162 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="c883ffae-9b9a-4622-9199-9aebffa4a5e6" containerName="istio-proxy" Oct 30 22:12:06.148264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.148176 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7443bc-8680-45f0-b375-2d1c7a8d6dac" containerName="istio-proxy" Oct 30 22:12:06.148264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.148187 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2b2ae6-303b-4cc1-ae81-c260f15f13ed" containerName="revoke-keys" Oct 30 22:12:06.148264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.148197 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="488cb661-6f7f-4ba3-b7d2-826dc69fcd94" containerName="istio-proxy" Oct 30 22:12:06.151671 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.151655 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.206036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.206007 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:12:06.211140 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.211110 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm"] Oct 30 22:12:06.235509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.235460 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.235702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.235637 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.235914 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.235892 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pswk\" (UniqueName: \"kubernetes.io/projected/656c87de-1ed6-4d3f-97db-12c3785e0ed7-kube-api-access-4pswk\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.260555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.257588 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:12:06.260795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.260766 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:12:06.260944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.260892 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:12:06.337582 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.337549 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.337772 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.337661 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.337920 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.337902 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pswk\" (UniqueName: \"kubernetes.io/projected/656c87de-1ed6-4d3f-97db-12c3785e0ed7-kube-api-access-4pswk\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.338469 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.338440 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.340244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.340223 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.400605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.400538 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pswk\" (UniqueName: \"kubernetes.io/projected/656c87de-1ed6-4d3f-97db-12c3785e0ed7-kube-api-access-4pswk\") pod \"maas-api-revoke-sub-5431d648-5gbfm\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.468012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.467979 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:06.896837 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:06.896808 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm"] Oct 30 22:12:06.897286 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:12:06.897263 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656c87de_1ed6_4d3f_97db_12c3785e0ed7.slice/crio-1106bd0dd3d8bcd35227fadbd31b6bab6ed95711aa0084afb03bbf34ca204350 WatchSource:0}: Error finding container 1106bd0dd3d8bcd35227fadbd31b6bab6ed95711aa0084afb03bbf34ca204350: Status 404 returned error can't find the container with id 1106bd0dd3d8bcd35227fadbd31b6bab6ed95711aa0084afb03bbf34ca204350 Oct 30 22:12:07.806749 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:07.806706 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" event={"ID":"656c87de-1ed6-4d3f-97db-12c3785e0ed7","Type":"ContainerStarted","Data":"e0f1a610e74bc288456b448553c1d759beeece942ffe7b2380b965bba263d622"} Oct 30 22:12:07.807154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:07.806802 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" event={"ID":"656c87de-1ed6-4d3f-97db-12c3785e0ed7","Type":"ContainerStarted","Data":"1106bd0dd3d8bcd35227fadbd31b6bab6ed95711aa0084afb03bbf34ca204350"} Oct 30 22:12:07.928654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:07.928597 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" podStartSLOduration=1.928583199 podStartE2EDuration="1.928583199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:07.926717076 +0000 UTC m=+1776.607866968" watchObservedRunningTime="2026-10-30 22:12:07.928583199 +0000 UTC m=+1776.609733091" Oct 30 22:12:08.815958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:08.815921 4933 generic.go:356] "Generic (PLEG): container finished" podID="656c87de-1ed6-4d3f-97db-12c3785e0ed7" containerID="e0f1a610e74bc288456b448553c1d759beeece942ffe7b2380b965bba263d622" exitCode=0 Oct 30 22:12:08.816438 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:08.815973 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" event={"ID":"656c87de-1ed6-4d3f-97db-12c3785e0ed7","Type":"ContainerDied","Data":"e0f1a610e74bc288456b448553c1d759beeece942ffe7b2380b965bba263d622"} Oct 30 22:12:10.007864 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.007839 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:10.103833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.103756 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-xks-ca\") pod \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " Oct 30 22:12:10.103991 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.103839 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-openshift-ca\") pod \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " Oct 30 22:12:10.103991 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.103880 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/656c87de-1ed6-4d3f-97db-12c3785e0ed7-kube-api-access-4pswk\" (UniqueName: \"kubernetes.io/projected/656c87de-1ed6-4d3f-97db-12c3785e0ed7-kube-api-access-4pswk\") pod \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\" (UID: \"656c87de-1ed6-4d3f-97db-12c3785e0ed7\") " Oct 30 22:12:10.104215 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.104190 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-openshift-ca" pod "656c87de-1ed6-4d3f-97db-12c3785e0ed7" (UID: "656c87de-1ed6-4d3f-97db-12c3785e0ed7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:10.104477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.104458 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:10.106124 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.106098 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-xks-ca" pod "656c87de-1ed6-4d3f-97db-12c3785e0ed7" (UID: "656c87de-1ed6-4d3f-97db-12c3785e0ed7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:10.106208 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.106187 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/656c87de-1ed6-4d3f-97db-12c3785e0ed7-kube-api-access-4pswk" pod "656c87de-1ed6-4d3f-97db-12c3785e0ed7" (UID: "656c87de-1ed6-4d3f-97db-12c3785e0ed7"). InnerVolumeSpecName "kube-api-access-4pswk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:10.205918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.205879 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/656c87de-1ed6-4d3f-97db-12c3785e0ed7-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:10.205918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.205911 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pswk\" (UniqueName: \"kubernetes.io/projected/656c87de-1ed6-4d3f-97db-12c3785e0ed7-kube-api-access-4pswk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:12:10.833735 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.833698 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" Oct 30 22:12:10.833735 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.833719 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm" event={"ID":"656c87de-1ed6-4d3f-97db-12c3785e0ed7","Type":"ContainerDied","Data":"1106bd0dd3d8bcd35227fadbd31b6bab6ed95711aa0084afb03bbf34ca204350"} Oct 30 22:12:10.833950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:10.833763 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1106bd0dd3d8bcd35227fadbd31b6bab6ed95711aa0084afb03bbf34ca204350" Oct 30 22:12:13.579892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:13.579857 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:12:27.059641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:27.059601 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc"] Oct 30 22:12:27.110139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:27.110108 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9421245-wfhtc"] Oct 30 22:12:27.468216 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:27.468133 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1c882f-bb5d-4af2-b539-ff75939dea39" path="/var/lib/kubelet/pods/2a1c882f-bb5d-4af2-b539-ff75939dea39/volumes" Oct 30 22:12:28.583755 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:12:28.583717 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:12:31.833284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:31.833255 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:12:31.840761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:31.840731 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:12:40.308277 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:12:40.308249 4933 scope.go:122] "RemoveContainer" containerID="ebe6b4fcf57af4ff0d6475f47c1cc1736168f85c83801dd826526a10354cfedd" Oct 30 22:12:58.549645 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:12:58.549603 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:13:23.051422 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:23.051340 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx"] Oct 30 22:13:23.099510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:23.099455 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f4ad945-9njnx"] Oct 30 22:13:23.470438 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:23.470367 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec1458e-b90b-4db9-8351-f0dbe457a113" path="/var/lib/kubelet/pods/dec1458e-b90b-4db9-8351-f0dbe457a113/volumes" Oct 30 22:13:28.426468 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:28.426437 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:13:28.545173 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:13:28.545134 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:13:40.346802 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:40.346771 4933 scope.go:122] "RemoveContainer" containerID="bef74aafdc27c06c1711affd540887e5a4b88f6df0c386b0eaaa758f5a8c2adc" Oct 30 22:13:40.359912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:40.359890 4933 scope.go:122] "RemoveContainer" containerID="d00fa0818f8886baf20b0e5885ab1745a097656772118b3e9eb71dc730db4c2f" Oct 30 22:13:40.370046 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:40.370028 4933 scope.go:122] "RemoveContainer" containerID="2a9b706b5b5230828fd05a5fe919f533586bf569a36fd927f467a8aaafa60214" Oct 30 22:13:40.383533 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:40.383510 4933 scope.go:122] "RemoveContainer" containerID="4bca01d9b264105f93f46034de36deb64cd6fe7c9c04babef9f21d5f2290bd62" Oct 30 22:13:45.992176 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:45.992138 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz"] Oct 30 22:13:45.993007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:45.992803 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" containerID="cri-o://741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f" gracePeriod=30 Oct 30 22:13:46.053641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:46.053603 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf"] Oct 30 22:13:46.054136 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:46.054099 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" containerID="cri-o://237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1" gracePeriod=30 Oct 30 22:13:47.851623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.851591 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb"] Oct 30 22:13:47.852205 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.852190 4933 cpu_manager.go:414] "containerMap: removing container" podUID="656c87de-1ed6-4d3f-97db-12c3785e0ed7" containerName="revoke-keys" Oct 30 22:13:47.852250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.852206 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="656c87de-1ed6-4d3f-97db-12c3785e0ed7" containerName="revoke-keys" Oct 30 22:13:47.852346 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.852336 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="656c87de-1ed6-4d3f-97db-12c3785e0ed7" containerName="revoke-keys" Oct 30 22:13:47.857627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.857609 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:47.912149 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.912129 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:47.912264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.912218 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:13:47.962805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.962756 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:13:47.963371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.963349 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:13:47.968006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:47.967981 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb"] Oct 30 22:13:48.002366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.002340 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.002627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.002610 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dpjb\" (UniqueName: \"kubernetes.io/projected/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-kube-api-access-5dpjb\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.002696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.002647 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.103591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.103528 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dpjb\" (UniqueName: \"kubernetes.io/projected/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-kube-api-access-5dpjb\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.103591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.103569 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.103751 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.103720 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.104332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.104297 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.106157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.106142 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.163091 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.163064 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dpjb\" (UniqueName: \"kubernetes.io/projected/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-kube-api-access-5dpjb\") pod \"maas-api-revoke-sub-4864ff62-dd7nb\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.175116 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.175095 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:48.572741 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:13:48.572707 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c1b1d1_374c_425e_b7dc_c8d95df3ed0a.slice/crio-201f7ebbc8757362633de14e4d4ae441d5985d587d3d2c26a7133b91d6e096b8 WatchSource:0}: Error finding container 201f7ebbc8757362633de14e4d4ae441d5985d587d3d2c26a7133b91d6e096b8: Status 404 returned error can't find the container with id 201f7ebbc8757362633de14e4d4ae441d5985d587d3d2c26a7133b91d6e096b8 Oct 30 22:13:48.574747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:48.574710 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb"] Oct 30 22:13:49.102234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:49.102144 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" event={"ID":"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a","Type":"ContainerStarted","Data":"01f7820053c151ab2662ddcb2da78585e67f6b92c50c3739ceb5ee05179e57dd"} Oct 30 22:13:49.102234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:49.102202 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" event={"ID":"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a","Type":"ContainerStarted","Data":"201f7ebbc8757362633de14e4d4ae441d5985d587d3d2c26a7133b91d6e096b8"} Oct 30 22:13:49.219744 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:49.219694 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" podStartSLOduration=2.219679208 podStartE2EDuration="2.219679208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:49.217328138 +0000 UTC m=+1877.898478031" watchObservedRunningTime="2026-10-30 22:13:49.219679208 +0000 UTC m=+1877.900829101" Oct 30 22:13:50.110818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:50.110780 4933 generic.go:356] "Generic (PLEG): container finished" podID="a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" containerID="01f7820053c151ab2662ddcb2da78585e67f6b92c50c3739ceb5ee05179e57dd" exitCode=0 Oct 30 22:13:50.111270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:50.110859 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" event={"ID":"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a","Type":"ContainerDied","Data":"01f7820053c151ab2662ddcb2da78585e67f6b92c50c3739ceb5ee05179e57dd"} Oct 30 22:13:51.369055 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.369013 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:13:51.416397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.416369 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:13:51.438876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.438853 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:51.459354 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459330 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-token\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-token\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459395 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-socket\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459512 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-envoy\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459578 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459555 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-podinfo\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459646 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-certs\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459699 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-kube-api-access-hzn6b\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-kube-api-access-hzn6b\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459712 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-socket" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.459818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459807 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-data\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459861 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-certs" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.459999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459869 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istiod-ca-cert\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.459999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.459981 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-credential-socket\") pod \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\" (UID: \"9531138b-2ae4-4de4-a05c-b88b2d69f20c\") " Oct 30 22:13:51.460389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460211 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-data" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.460453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460393 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-credential-socket" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.460724 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460692 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istiod-ca-cert" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:51.460908 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460885 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.460993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460915 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.460993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460931 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.460993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460947 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.460993 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.460961 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.461865 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.461841 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-envoy" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.461987 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.461966 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-podinfo" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:51.462184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.462162 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-token" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:51.462352 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.462335 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-kube-api-access-hzn6b" pod "9531138b-2ae4-4de4-a05c-b88b2d69f20c" (UID: "9531138b-2ae4-4de4-a05c-b88b2d69f20c"). InnerVolumeSpecName "kube-api-access-hzn6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:51.561776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.561722 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-token\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-token\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.561776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.561752 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-podinfo\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.561954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.561792 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-openshift-ca\") pod \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " Oct 30 22:13:51.561954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.561832 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-credential-socket\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.561954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.561866 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-kube-api-access-ml7d7\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-kube-api-access-ml7d7\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.561954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.561901 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-envoy\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.562187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562003 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-xks-ca\") pod \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " Oct 30 22:13:51.562187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562034 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istiod-ca-cert\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.562187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562063 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-socket\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.562187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562115 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-credential-socket" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.562187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562143 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-kube-api-access-5dpjb\" (UniqueName: \"kubernetes.io/projected/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-kube-api-access-5dpjb\") pod \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\" (UID: \"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a\") " Oct 30 22:13:51.562187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562178 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-openshift-ca" pod "a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" (UID: "a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:51.562187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562185 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-data\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.562577 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562248 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-certs\") pod \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\" (UID: \"405d6876-36a7-4faa-92a3-3ddc9bd511a1\") " Oct 30 22:13:51.562643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562584 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-certs" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562756 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-socket" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.562794 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istiod-ca-cert" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563079 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-data" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563307 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563329 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563345 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563360 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563375 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563388 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563401 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9531138b-2ae4-4de4-a05c-b88b2d69f20c-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.563413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563415 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzn6b\" (UniqueName: \"kubernetes.io/projected/9531138b-2ae4-4de4-a05c-b88b2d69f20c-kube-api-access-hzn6b\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.564315 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563429 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.564315 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.563441 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.564670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.564650 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-token" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:51.564832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.564752 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-podinfo" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:13:51.564832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.564766 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-xks-ca" pod "a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" (UID: "a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:51.565128 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.565111 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-kube-api-access-ml7d7" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "kube-api-access-ml7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:51.565171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.565138 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-kube-api-access-5dpjb" pod "a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" (UID: "a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a"). InnerVolumeSpecName "kube-api-access-5dpjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:51.565289 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.565269 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-envoy" pod "405d6876-36a7-4faa-92a3-3ddc9bd511a1" (UID: "405d6876-36a7-4faa-92a3-3ddc9bd511a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:13:51.668554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.668522 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.668554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.668556 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.668798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.668570 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml7d7\" (UniqueName: \"kubernetes.io/projected/405d6876-36a7-4faa-92a3-3ddc9bd511a1-kube-api-access-ml7d7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.668798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.668584 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/405d6876-36a7-4faa-92a3-3ddc9bd511a1-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.668798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.668596 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:51.668798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:51.668610 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dpjb\" (UniqueName: \"kubernetes.io/projected/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a-kube-api-access-5dpjb\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:13:52.128312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.128281 4933 generic.go:356] "Generic (PLEG): container finished" podID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerID="237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1" exitCode=0 Oct 30 22:13:52.128527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.128339 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" Oct 30 22:13:52.128527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.128370 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" event={"ID":"9531138b-2ae4-4de4-a05c-b88b2d69f20c","Type":"ContainerDied","Data":"237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1"} Oct 30 22:13:52.128527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.128422 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf" event={"ID":"9531138b-2ae4-4de4-a05c-b88b2d69f20c","Type":"ContainerDied","Data":"66fb2bd463784215d7298e2047dd8a8d44a8d544a41c0e5ca567bb3fb6152295"} Oct 30 22:13:52.128527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.128444 4933 scope.go:122] "RemoveContainer" containerID="237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1" Oct 30 22:13:52.130024 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.130003 4933 generic.go:356] "Generic (PLEG): container finished" podID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerID="741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f" exitCode=0 Oct 30 22:13:52.130121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.130056 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" Oct 30 22:13:52.130121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.130096 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" event={"ID":"405d6876-36a7-4faa-92a3-3ddc9bd511a1","Type":"ContainerDied","Data":"741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f"} Oct 30 22:13:52.130225 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.130143 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz" event={"ID":"405d6876-36a7-4faa-92a3-3ddc9bd511a1","Type":"ContainerDied","Data":"f442450e5e169de10a99b89c874667d5f1ba7d20b63d46830eaf33530b1d9697"} Oct 30 22:13:52.131864 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.131838 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" event={"ID":"a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a","Type":"ContainerDied","Data":"201f7ebbc8757362633de14e4d4ae441d5985d587d3d2c26a7133b91d6e096b8"} Oct 30 22:13:52.131976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.131877 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb" Oct 30 22:13:52.131976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.131884 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="201f7ebbc8757362633de14e4d4ae441d5985d587d3d2c26a7133b91d6e096b8" Oct 30 22:13:52.142400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.142382 4933 scope.go:122] "RemoveContainer" containerID="237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1" Oct 30 22:13:52.142723 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:13:52.142703 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1\": container with ID starting with 237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1 not found: ID does not exist" containerID="237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1" Oct 30 22:13:52.142807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.142731 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1"} err="failed to get container status \"237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1\": rpc error: code = NotFound desc = could not find container \"237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1\": container with ID starting with 237dec77597fdd9e0adca1f5d6126a8004511d30b986bbb87e799796420ac4f1 not found: ID does not exist" Oct 30 22:13:52.142807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.142750 4933 scope.go:122] "RemoveContainer" containerID="741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f" Oct 30 22:13:52.152054 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.152036 4933 scope.go:122] "RemoveContainer" containerID="741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f" Oct 30 22:13:52.152309 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:13:52.152294 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f\": container with ID starting with 741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f not found: ID does not exist" containerID="741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f" Oct 30 22:13:52.152388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.152315 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f"} err="failed to get container status \"741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f\": rpc error: code = NotFound desc = could not find container \"741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f\": container with ID starting with 741eabfeffbc74792c1fa0b823bb099b97594424caac5b2b1e26edc0e1aa9e3f not found: ID does not exist" Oct 30 22:13:52.353540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.353444 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz"] Oct 30 22:13:52.359190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.359167 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-7dbgz"] Oct 30 22:13:52.469173 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.469141 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf"] Oct 30 22:13:52.487478 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:52.487449 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dde1e8-openshift-default-7c667f55ff-2wgxf"] Oct 30 22:13:53.467396 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:53.467363 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" path="/var/lib/kubelet/pods/405d6876-36a7-4faa-92a3-3ddc9bd511a1/volumes" Oct 30 22:13:53.467885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:53.467869 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" path="/var/lib/kubelet/pods/9531138b-2ae4-4de4-a05c-b88b2d69f20c/volumes" Oct 30 22:13:58.020319 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.020286 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs"] Oct 30 22:13:58.020951 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.020933 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" Oct 30 22:13:58.020951 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.020948 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" Oct 30 22:13:58.021057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.020957 4933 cpu_manager.go:414] "containerMap: removing container" podUID="a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" containerName="revoke-keys" Oct 30 22:13:58.021057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.020962 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" containerName="revoke-keys" Oct 30 22:13:58.021057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.020974 4933 cpu_manager.go:414] "containerMap: removing container" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" Oct 30 22:13:58.021057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.020978 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" Oct 30 22:13:58.021179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.021066 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9531138b-2ae4-4de4-a05c-b88b2d69f20c" containerName="istio-proxy" Oct 30 22:13:58.021179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.021075 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" containerName="revoke-keys" Oct 30 22:13:58.021179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.021082 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="405d6876-36a7-4faa-92a3-3ddc9bd511a1" containerName="istio-proxy" Oct 30 22:13:58.026267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.026242 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.084478 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.084448 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs"] Oct 30 22:13:58.127298 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.127271 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.127449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.127389 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.127563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.127541 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbl4\" (UniqueName: \"kubernetes.io/projected/31c495e1-0bc0-430e-9ddf-c6a959d4c624-kube-api-access-6cbl4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.128946 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.128931 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:13:58.130782 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.130768 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:58.130782 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.130779 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:13:58.130880 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.130808 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:13:58.228223 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.228200 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.228346 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.228267 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbl4\" (UniqueName: \"kubernetes.io/projected/31c495e1-0bc0-430e-9ddf-c6a959d4c624-kube-api-access-6cbl4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.228393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.228350 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.228972 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.228956 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.230746 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.230729 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.286405 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.286385 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbl4\" (UniqueName: \"kubernetes.io/projected/31c495e1-0bc0-430e-9ddf-c6a959d4c624-kube-api-access-6cbl4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.339413 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.339393 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:13:58.547088 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:13:58.547010 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:13:58.753025 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:58.752996 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs"] Oct 30 22:13:59.249611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:59.249527 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" event={"ID":"31c495e1-0bc0-430e-9ddf-c6a959d4c624","Type":"ContainerStarted","Data":"96f06bfa9b038f50748892e9f19ec502bb75c7f0725a25c6a89c9500b2ec7927"} Oct 30 22:13:59.249611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:59.249592 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" event={"ID":"31c495e1-0bc0-430e-9ddf-c6a959d4c624","Type":"ContainerStarted","Data":"78a475a21e10bf804ef5741d1dd41b14a515bbfc892047fe3ed6b939ef858b32"} Oct 30 22:13:59.368882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:13:59.368834 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" podStartSLOduration=2.36882172 podStartE2EDuration="2.36882172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:59.365019151 +0000 UTC m=+1888.046169046" watchObservedRunningTime="2026-10-30 22:13:59.36882172 +0000 UTC m=+1888.049971612" Oct 30 22:14:00.258863 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:00.258832 4933 generic.go:356] "Generic (PLEG): container finished" podID="31c495e1-0bc0-430e-9ddf-c6a959d4c624" containerID="96f06bfa9b038f50748892e9f19ec502bb75c7f0725a25c6a89c9500b2ec7927" exitCode=0 Oct 30 22:14:00.259302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:00.258930 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" event={"ID":"31c495e1-0bc0-430e-9ddf-c6a959d4c624","Type":"ContainerDied","Data":"96f06bfa9b038f50748892e9f19ec502bb75c7f0725a25c6a89c9500b2ec7927"} Oct 30 22:14:01.615048 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.615013 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:14:01.717730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.717698 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-xks-ca\") pod \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " Oct 30 22:14:01.717888 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.717812 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-openshift-ca\") pod \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " Oct 30 22:14:01.717944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.717913 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c495e1-0bc0-430e-9ddf-c6a959d4c624-kube-api-access-6cbl4\" (UniqueName: \"kubernetes.io/projected/31c495e1-0bc0-430e-9ddf-c6a959d4c624-kube-api-access-6cbl4\") pod \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\" (UID: \"31c495e1-0bc0-430e-9ddf-c6a959d4c624\") " Oct 30 22:14:01.718255 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.718204 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-openshift-ca" pod "31c495e1-0bc0-430e-9ddf-c6a959d4c624" (UID: "31c495e1-0bc0-430e-9ddf-c6a959d4c624"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:01.718577 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.718552 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.719954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.719931 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-xks-ca" pod "31c495e1-0bc0-430e-9ddf-c6a959d4c624" (UID: "31c495e1-0bc0-430e-9ddf-c6a959d4c624"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:01.720110 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.720096 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c495e1-0bc0-430e-9ddf-c6a959d4c624-kube-api-access-6cbl4" pod "31c495e1-0bc0-430e-9ddf-c6a959d4c624" (UID: "31c495e1-0bc0-430e-9ddf-c6a959d4c624"). InnerVolumeSpecName "kube-api-access-6cbl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:01.820233 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.820199 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbl4\" (UniqueName: \"kubernetes.io/projected/31c495e1-0bc0-430e-9ddf-c6a959d4c624-kube-api-access-6cbl4\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:01.820233 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:01.820234 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31c495e1-0bc0-430e-9ddf-c6a959d4c624-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:02.302639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:02.302612 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" Oct 30 22:14:02.302852 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:02.302649 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs" event={"ID":"31c495e1-0bc0-430e-9ddf-c6a959d4c624","Type":"ContainerDied","Data":"78a475a21e10bf804ef5741d1dd41b14a515bbfc892047fe3ed6b939ef858b32"} Oct 30 22:14:02.302852 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:02.302695 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78a475a21e10bf804ef5741d1dd41b14a515bbfc892047fe3ed6b939ef858b32" Oct 30 22:14:17.124473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:17.124441 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:14:28.549536 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:14:28.545641 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:14:33.268954 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.268916 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4"] Oct 30 22:14:33.269753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.269732 4933 cpu_manager.go:414] "containerMap: removing container" podUID="31c495e1-0bc0-430e-9ddf-c6a959d4c624" containerName="revoke-keys" Oct 30 22:14:33.269753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.269752 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c495e1-0bc0-430e-9ddf-c6a959d4c624" containerName="revoke-keys" Oct 30 22:14:33.269976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.269961 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c495e1-0bc0-430e-9ddf-c6a959d4c624" containerName="revoke-keys" Oct 30 22:14:33.273664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.273644 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.330004 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.329974 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4"] Oct 30 22:14:33.372452 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.372414 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwksv\" (UniqueName: \"kubernetes.io/projected/3deada50-f885-44c5-a5b8-e46727f7b606-kube-api-access-jwksv\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.372667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.372643 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.372776 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.372760 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.375030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.375011 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:33.375143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.375037 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:33.425661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.425592 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:33.425661 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.425629 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:14:33.473964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.473940 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.474126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.474032 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.474206 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.474138 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwksv\" (UniqueName: \"kubernetes.io/projected/3deada50-f885-44c5-a5b8-e46727f7b606-kube-api-access-jwksv\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.474625 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.474602 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.476721 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.476705 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.537786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.537761 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwksv\" (UniqueName: \"kubernetes.io/projected/3deada50-f885-44c5-a5b8-e46727f7b606-kube-api-access-jwksv\") pod \"maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.587677 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.587651 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:33.965717 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:14:33.965688 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3deada50_f885_44c5_a5b8_e46727f7b606.slice/crio-7afa6309c15468229abe687c8f889995816ddd5216707b24b043e28418295781 WatchSource:0}: Error finding container 7afa6309c15468229abe687c8f889995816ddd5216707b24b043e28418295781: Status 404 returned error can't find the container with id 7afa6309c15468229abe687c8f889995816ddd5216707b24b043e28418295781 Oct 30 22:14:33.966792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:33.966767 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4"] Oct 30 22:14:34.751061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:34.751021 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" event={"ID":"3deada50-f885-44c5-a5b8-e46727f7b606","Type":"ContainerStarted","Data":"706abb3d6a1978ff94075525aa447a441234a0f4ea7ccc53e18176d6ee710ba6"} Oct 30 22:14:34.751576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:34.751085 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" event={"ID":"3deada50-f885-44c5-a5b8-e46727f7b606","Type":"ContainerStarted","Data":"7afa6309c15468229abe687c8f889995816ddd5216707b24b043e28418295781"} Oct 30 22:14:34.871427 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:34.871376 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" podStartSLOduration=1.871362431 podStartE2EDuration="1.871362431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:34.868462196 +0000 UTC m=+1923.549612133" watchObservedRunningTime="2026-10-30 22:14:34.871362431 +0000 UTC m=+1923.552512324" Oct 30 22:14:35.760163 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:35.760125 4933 generic.go:356] "Generic (PLEG): container finished" podID="3deada50-f885-44c5-a5b8-e46727f7b606" containerID="706abb3d6a1978ff94075525aa447a441234a0f4ea7ccc53e18176d6ee710ba6" exitCode=0 Oct 30 22:14:35.760596 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:35.760206 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" event={"ID":"3deada50-f885-44c5-a5b8-e46727f7b606","Type":"ContainerDied","Data":"706abb3d6a1978ff94075525aa447a441234a0f4ea7ccc53e18176d6ee710ba6"} Oct 30 22:14:36.974400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:36.974370 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:37.018007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.017984 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-xks-ca\") pod \"3deada50-f885-44c5-a5b8-e46727f7b606\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " Oct 30 22:14:37.018141 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.018018 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3deada50-f885-44c5-a5b8-e46727f7b606-kube-api-access-jwksv\" (UniqueName: \"kubernetes.io/projected/3deada50-f885-44c5-a5b8-e46727f7b606-kube-api-access-jwksv\") pod \"3deada50-f885-44c5-a5b8-e46727f7b606\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " Oct 30 22:14:37.018141 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.018108 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-openshift-ca\") pod \"3deada50-f885-44c5-a5b8-e46727f7b606\" (UID: \"3deada50-f885-44c5-a5b8-e46727f7b606\") " Oct 30 22:14:37.018469 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.018442 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-openshift-ca" pod "3deada50-f885-44c5-a5b8-e46727f7b606" (UID: "3deada50-f885-44c5-a5b8-e46727f7b606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:37.018778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.018760 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:37.020305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.020255 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-xks-ca" pod "3deada50-f885-44c5-a5b8-e46727f7b606" (UID: "3deada50-f885-44c5-a5b8-e46727f7b606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:37.020397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.020380 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3deada50-f885-44c5-a5b8-e46727f7b606-kube-api-access-jwksv" pod "3deada50-f885-44c5-a5b8-e46727f7b606" (UID: "3deada50-f885-44c5-a5b8-e46727f7b606"). InnerVolumeSpecName "kube-api-access-jwksv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:37.119944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.119917 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3deada50-f885-44c5-a5b8-e46727f7b606-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:37.119944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.119944 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwksv\" (UniqueName: \"kubernetes.io/projected/3deada50-f885-44c5-a5b8-e46727f7b606-kube-api-access-jwksv\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:14:37.815258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.815221 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" event={"ID":"3deada50-f885-44c5-a5b8-e46727f7b606","Type":"ContainerDied","Data":"7afa6309c15468229abe687c8f889995816ddd5216707b24b043e28418295781"} Oct 30 22:14:37.815443 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.815263 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4" Oct 30 22:14:37.815443 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:37.815299 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7afa6309c15468229abe687c8f889995816ddd5216707b24b043e28418295781" Oct 30 22:14:40.424772 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:40.424743 4933 scope.go:122] "RemoveContainer" containerID="cf6518836ff86f5abfd5c969d840049f0854466a43e6622f4d0ec90e04d2bedd" Oct 30 22:14:49.714354 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:49.714322 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs"] Oct 30 22:14:49.765144 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:49.765101 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2a25a4-65zvs"] Oct 30 22:14:51.476265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:51.476229 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c495e1-0bc0-430e-9ddf-c6a959d4c624" path="/var/lib/kubelet/pods/31c495e1-0bc0-430e-9ddf-c6a959d4c624/volumes" Oct 30 22:14:57.726862 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.726828 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm"] Oct 30 22:14:57.727595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.727575 4933 cpu_manager.go:414] "containerMap: removing container" podUID="3deada50-f885-44c5-a5b8-e46727f7b606" containerName="revoke-keys" Oct 30 22:14:57.727672 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.727598 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3deada50-f885-44c5-a5b8-e46727f7b606" containerName="revoke-keys" Oct 30 22:14:57.727773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.727762 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="3deada50-f885-44c5-a5b8-e46727f7b606" containerName="revoke-keys" Oct 30 22:14:57.731812 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.731789 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.791061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.791032 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm"] Oct 30 22:14:57.834614 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.834585 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:57.834758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.834622 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:57.834758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.834678 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:14:57.838375 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.838347 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dksw\" (UniqueName: \"kubernetes.io/projected/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-kube-api-access-4dksw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.838503 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.838409 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.838571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.838553 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.841919 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.841902 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:57.940202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.940166 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.940449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.940428 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dksw\" (UniqueName: \"kubernetes.io/projected/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-kube-api-access-4dksw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.940575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.940543 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.940975 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.940955 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.943104 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.943082 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:57.999652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:57.999599 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dksw\" (UniqueName: \"kubernetes.io/projected/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-kube-api-access-4dksw\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:58.049195 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:58.049164 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:14:58.442239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:58.442207 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm"] Oct 30 22:14:58.442377 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:14:58.442312 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bc70b4_aaa3_482e_bd60_08ec61c2e18c.slice/crio-58d204c9a442567cf5015da26def7549fdb0c8b9413bf8d33f0e4471563e84f3 WatchSource:0}: Error finding container 58d204c9a442567cf5015da26def7549fdb0c8b9413bf8d33f0e4471563e84f3: Status 404 returned error can't find the container with id 58d204c9a442567cf5015da26def7549fdb0c8b9413bf8d33f0e4471563e84f3 Oct 30 22:14:58.546593 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:14:58.546562 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:14:59.037040 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:59.037007 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" event={"ID":"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c","Type":"ContainerStarted","Data":"7710609c1c3ddfd4d38d485b89e082359f1d89f73ccf642a25fb1ad19154ab97"} Oct 30 22:14:59.037417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:59.037059 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" event={"ID":"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c","Type":"ContainerStarted","Data":"58d204c9a442567cf5015da26def7549fdb0c8b9413bf8d33f0e4471563e84f3"} Oct 30 22:14:59.158902 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:14:59.158844 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" podStartSLOduration=2.158824934 podStartE2EDuration="2.158824934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:59.155694835 +0000 UTC m=+1947.836844729" watchObservedRunningTime="2026-10-30 22:14:59.158824934 +0000 UTC m=+1947.839974825" Oct 30 22:15:00.050077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.050040 4933 generic.go:356] "Generic (PLEG): container finished" podID="a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" containerID="7710609c1c3ddfd4d38d485b89e082359f1d89f73ccf642a25fb1ad19154ab97" exitCode=0 Oct 30 22:15:00.050547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.050120 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" event={"ID":"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c","Type":"ContainerDied","Data":"7710609c1c3ddfd4d38d485b89e082359f1d89f73ccf642a25fb1ad19154ab97"} Oct 30 22:15:00.180917 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.180884 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq"] Oct 30 22:15:00.186032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.186012 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.231646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.231614 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r"] Oct 30 22:15:00.247395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.247365 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2"] Oct 30 22:15:00.247558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.247529 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.252146 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.252122 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm"] Oct 30 22:15:00.252277 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.252256 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.256354 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.256337 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.258246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.258203 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc"] Oct 30 22:15:00.263235 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.263216 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq"] Oct 30 22:15:00.263467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.263337 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.271984 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.271954 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.272243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.272224 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fc4g\" (UniqueName: \"kubernetes.io/projected/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-kube-api-access-7fc4g\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.273765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.273744 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh"] Oct 30 22:15:00.279755 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.279179 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.279755 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.279189 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8"] Oct 30 22:15:00.285290 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.285270 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.305967 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.305944 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r"] Oct 30 22:15:00.319278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.318414 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8"] Oct 30 22:15:00.325908 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.325884 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:00.335639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.335614 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm"] Oct 30 22:15:00.370804 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.370774 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh"] Oct 30 22:15:00.378351 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.378320 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc"] Oct 30 22:15:00.382410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.382348 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx"] Oct 30 22:15:00.386885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.386800 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7bf9c8-7c39-417b-937e-63516a76f54a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.386885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.386837 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsx9\" (UniqueName: \"kubernetes.io/projected/76c59b74-a98f-4464-b7e0-a228677cea1d-kube-api-access-rvsx9\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.387092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.386929 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31c4d05-a234-4df5-a548-0574613c51f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.387092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.386979 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c59b74-a98f-4464-b7e0-a228677cea1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.387092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387056 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d83313-5765-48db-9683-c852af36ad58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.387271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387101 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fc4g\" (UniqueName: \"kubernetes.io/projected/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-kube-api-access-7fc4g\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.387271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387176 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54d4\" (UniqueName: \"kubernetes.io/projected/e31c4d05-a234-4df5-a548-0574613c51f6-kube-api-access-k54d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.387271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387220 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljpw\" (UniqueName: \"kubernetes.io/projected/4e7bf9c8-7c39-417b-937e-63516a76f54a-kube-api-access-8ljpw\") pod \"maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.387271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387249 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c1f3f-7e42-488c-ba8c-36342be21470-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-hghgh\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.387912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387282 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0408ead9-3525-4ef7-a92f-17189843b02f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.387912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387351 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.387912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387429 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhfnk\" (UniqueName: \"kubernetes.io/projected/031c1f3f-7e42-488c-ba8c-36342be21470-kube-api-access-mhfnk\") pod \"maas-api-key-cleanup-29889975-hghgh\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.387912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387462 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ncx9\" (UniqueName: \"kubernetes.io/projected/0408ead9-3525-4ef7-a92f-17189843b02f-kube-api-access-2ncx9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.387912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.387544 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzglx\" (UniqueName: \"kubernetes.io/projected/f8d83313-5765-48db-9683-c852af36ad58-kube-api-access-vzglx\") pod \"maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.388250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.388231 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2"] Oct 30 22:15:00.388361 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.388347 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.388970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.388948 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.391088 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.391067 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8"] Oct 30 22:15:00.405382 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.405358 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8"] Oct 30 22:15:00.437411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.434950 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx"] Oct 30 22:15:00.449125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.449098 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fc4g\" (UniqueName: \"kubernetes.io/projected/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-kube-api-access-7fc4g\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.489117 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489090 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1327a728-8c81-4c47-9985-397ff7342c4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.489250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489129 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d83313-5765-48db-9683-c852af36ad58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.489250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489228 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54d4\" (UniqueName: \"kubernetes.io/projected/e31c4d05-a234-4df5-a548-0574613c51f6-kube-api-access-k54d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489292 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljpw\" (UniqueName: \"kubernetes.io/projected/4e7bf9c8-7c39-417b-937e-63516a76f54a-kube-api-access-8ljpw\") pod \"maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489323 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c1f3f-7e42-488c-ba8c-36342be21470-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-hghgh\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489360 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0408ead9-3525-4ef7-a92f-17189843b02f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489410 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489473 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbxr\" (UniqueName: \"kubernetes.io/projected/1327a728-8c81-4c47-9985-397ff7342c4b-kube-api-access-mdbxr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489549 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhfnk\" (UniqueName: \"kubernetes.io/projected/031c1f3f-7e42-488c-ba8c-36342be21470-kube-api-access-mhfnk\") pod \"maas-api-key-cleanup-29889975-hghgh\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489586 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ncx9\" (UniqueName: \"kubernetes.io/projected/0408ead9-3525-4ef7-a92f-17189843b02f-kube-api-access-2ncx9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.489709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489640 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzglx\" (UniqueName: \"kubernetes.io/projected/f8d83313-5765-48db-9683-c852af36ad58-kube-api-access-vzglx\") pod \"maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.490172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489728 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7bf9c8-7c39-417b-937e-63516a76f54a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.490172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489746 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsx9\" (UniqueName: \"kubernetes.io/projected/76c59b74-a98f-4464-b7e0-a228677cea1d-kube-api-access-rvsx9\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.490172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489798 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjwb\" (UniqueName: \"kubernetes.io/projected/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-kube-api-access-6zjwb\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:00.490172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489829 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d83313-5765-48db-9683-c852af36ad58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.490172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489829 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31c4d05-a234-4df5-a548-0574613c51f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.490172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.489893 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c59b74-a98f-4464-b7e0-a228677cea1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.490476 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.490257 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0408ead9-3525-4ef7-a92f-17189843b02f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.490476 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.490383 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31c4d05-a234-4df5-a548-0574613c51f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.490476 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.490436 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c59b74-a98f-4464-b7e0-a228677cea1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.490750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.490726 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c1f3f-7e42-488c-ba8c-36342be21470-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-hghgh\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.490999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.490977 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7bf9c8-7c39-417b-937e-63516a76f54a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.502127 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.502103 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:00.549242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.549212 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ncx9\" (UniqueName: \"kubernetes.io/projected/0408ead9-3525-4ef7-a92f-17189843b02f-kube-api-access-2ncx9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.549385 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.549252 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljpw\" (UniqueName: \"kubernetes.io/projected/4e7bf9c8-7c39-417b-937e-63516a76f54a-kube-api-access-8ljpw\") pod \"maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.549941 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.549917 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhfnk\" (UniqueName: \"kubernetes.io/projected/031c1f3f-7e42-488c-ba8c-36342be21470-kube-api-access-mhfnk\") pod \"maas-api-key-cleanup-29889975-hghgh\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.550045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.549961 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzglx\" (UniqueName: \"kubernetes.io/projected/f8d83313-5765-48db-9683-c852af36ad58-kube-api-access-vzglx\") pod \"maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.550371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.550354 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54d4\" (UniqueName: \"kubernetes.io/projected/e31c4d05-a234-4df5-a548-0574613c51f6-kube-api-access-k54d4\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.554133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.554107 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsx9\" (UniqueName: \"kubernetes.io/projected/76c59b74-a98f-4464-b7e0-a228677cea1d-kube-api-access-rvsx9\") pod \"maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.565384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.565038 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:00.575529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.575482 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.587688 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.594447 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1327a728-8c81-4c47-9985-397ff7342c4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.594607 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.594670 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbxr\" (UniqueName: \"kubernetes.io/projected/1327a728-8c81-4c47-9985-397ff7342c4b-kube-api-access-mdbxr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.595905 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.597016 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.597286 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1327a728-8c81-4c47-9985-397ff7342c4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.600726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.598332 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjwb\" (UniqueName: \"kubernetes.io/projected/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-kube-api-access-6zjwb\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:00.605850 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.605830 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:00.620667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.620645 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:00.655097 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.655008 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbxr\" (UniqueName: \"kubernetes.io/projected/1327a728-8c81-4c47-9985-397ff7342c4b-kube-api-access-mdbxr\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.659609 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.659577 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjwb\" (UniqueName: \"kubernetes.io/projected/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-kube-api-access-6zjwb\") pod \"maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:00.713039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.712780 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:00.904393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.904364 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq"] Oct 30 22:15:00.944211 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:00.944182 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:01.074939 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.073507 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" event={"ID":"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f","Type":"ContainerStarted","Data":"00219bf6002d0529807e39df5e1f7fb3d9e7d0ecad04bdcd2ec9f4c46c1179d2"} Oct 30 22:15:01.125440 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.125399 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh"] Oct 30 22:15:01.126978 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:01.126952 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031c1f3f_7e42_488c_ba8c_36342be21470.slice/crio-f5a766a05c86c2c847fea4df99dccaf6d3f56f9abe76879e0c650b843a0ae958 WatchSource:0}: Error finding container f5a766a05c86c2c847fea4df99dccaf6d3f56f9abe76879e0c650b843a0ae958: Status 404 returned error can't find the container with id f5a766a05c86c2c847fea4df99dccaf6d3f56f9abe76879e0c650b843a0ae958 Oct 30 22:15:01.201525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.199041 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx"] Oct 30 22:15:01.210823 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.209812 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r"] Oct 30 22:15:01.210823 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:01.209963 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1327a728_8c81_4c47_9985_397ff7342c4b.slice/crio-db592af9ec70697547e28860e8e2a95b705d742f0ec4a5a669eb2ed445825db5 WatchSource:0}: Error finding container db592af9ec70697547e28860e8e2a95b705d742f0ec4a5a669eb2ed445825db5: Status 404 returned error can't find the container with id db592af9ec70697547e28860e8e2a95b705d742f0ec4a5a669eb2ed445825db5 Oct 30 22:15:01.213305 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:01.213284 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c59b74_a98f_4464_b7e0_a228677cea1d.slice/crio-081ceff5a85c88b191fc5e235f47ac1e4605dae6bf451d58dba4289588e6c86c WatchSource:0}: Error finding container 081ceff5a85c88b191fc5e235f47ac1e4605dae6bf451d58dba4289588e6c86c: Status 404 returned error can't find the container with id 081ceff5a85c88b191fc5e235f47ac1e4605dae6bf451d58dba4289588e6c86c Oct 30 22:15:01.263450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.263421 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2"] Oct 30 22:15:01.267997 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:01.265586 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31c4d05_a234_4df5_a548_0574613c51f6.slice/crio-bc3d70084d8c15a48476aa0db2a8f1fd6a54eef032b1bacef84235597c28f689 WatchSource:0}: Error finding container bc3d70084d8c15a48476aa0db2a8f1fd6a54eef032b1bacef84235597c28f689: Status 404 returned error can't find the container with id bc3d70084d8c15a48476aa0db2a8f1fd6a54eef032b1bacef84235597c28f689 Oct 30 22:15:01.267997 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.266723 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm"] Oct 30 22:15:01.282162 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.281920 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:15:01.282162 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.281959 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc"] Oct 30 22:15:01.286955 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:01.286920 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d83313_5765_48db_9683_c852af36ad58.slice/crio-2c274ba4e270e53a1737f578338758d9a3c9660df9e6e24c104e3e3be2587748 WatchSource:0}: Error finding container 2c274ba4e270e53a1737f578338758d9a3c9660df9e6e24c104e3e3be2587748: Status 404 returned error can't find the container with id 2c274ba4e270e53a1737f578338758d9a3c9660df9e6e24c104e3e3be2587748 Oct 30 22:15:01.346893 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.346864 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8"] Oct 30 22:15:01.347432 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:01.347409 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea06c18b_0aeb_4d30_a27f_9b08ca70ed72.slice/crio-b9018bb34b07ca7a7b45c69cba7b9650ba64fbce9eb4464df8260faf25ce7a5f WatchSource:0}: Error finding container b9018bb34b07ca7a7b45c69cba7b9650ba64fbce9eb4464df8260faf25ce7a5f: Status 404 returned error can't find the container with id b9018bb34b07ca7a7b45c69cba7b9650ba64fbce9eb4464df8260faf25ce7a5f Oct 30 22:15:01.369849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.369827 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8"] Oct 30 22:15:01.370571 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:01.370546 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0408ead9_3525_4ef7_a92f_17189843b02f.slice/crio-912450d3001dff5a133609280778fdc66b99dd112738ed982864ccad1068ffa4 WatchSource:0}: Error finding container 912450d3001dff5a133609280778fdc66b99dd112738ed982864ccad1068ffa4: Status 404 returned error can't find the container with id 912450d3001dff5a133609280778fdc66b99dd112738ed982864ccad1068ffa4 Oct 30 22:15:01.433045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.433009 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-kube-api-access-4dksw\" (UniqueName: \"kubernetes.io/projected/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-kube-api-access-4dksw\") pod \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " Oct 30 22:15:01.433664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.433639 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-openshift-ca\") pod \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " Oct 30 22:15:01.433803 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.433769 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-xks-ca\") pod \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\" (UID: \"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c\") " Oct 30 22:15:01.435805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.435766 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-openshift-ca" pod "a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" (UID: "a0bc70b4-aaa3-482e-bd60-08ec61c2e18c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:01.533632 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.532686 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-kube-api-access-4dksw" pod "a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" (UID: "a0bc70b4-aaa3-482e-bd60-08ec61c2e18c"). InnerVolumeSpecName "kube-api-access-4dksw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:01.534554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.534517 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-xks-ca" pod "a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" (UID: "a0bc70b4-aaa3-482e-bd60-08ec61c2e18c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:01.536943 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.536916 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dksw\" (UniqueName: \"kubernetes.io/projected/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-kube-api-access-4dksw\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:01.537046 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.536948 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:01.537046 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:01.536963 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:02.087244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.087205 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerStarted","Data":"a056f603ebbb945a85920f26860ce9b686e7fa91b22b4669b88bd83388b37e0e"} Oct 30 22:15:02.087742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.087269 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerStarted","Data":"f5a766a05c86c2c847fea4df99dccaf6d3f56f9abe76879e0c650b843a0ae958"} Oct 30 22:15:02.089107 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.089078 4933 generic.go:356] "Generic (PLEG): container finished" podID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerID="69a4b77de1b721a8c22d001635243ee5975bf1b09892eb7003251adbb49258a8" exitCode=60 Oct 30 22:15:02.089232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.089165 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" event={"ID":"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f","Type":"ContainerDied","Data":"69a4b77de1b721a8c22d001635243ee5975bf1b09892eb7003251adbb49258a8"} Oct 30 22:15:02.089382 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.089367 4933 scope.go:122] "RemoveContainer" containerID="69a4b77de1b721a8c22d001635243ee5975bf1b09892eb7003251adbb49258a8" Oct 30 22:15:02.091219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.091199 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerStarted","Data":"8aa3be71ac92f2f6334d1c79d5fc692673b33376b4f83a9cebd6659ce8f8fa98"} Oct 30 22:15:02.091309 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.091244 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerStarted","Data":"bc3d70084d8c15a48476aa0db2a8f1fd6a54eef032b1bacef84235597c28f689"} Oct 30 22:15:02.093418 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.093393 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerStarted","Data":"f28311e04131359368ed10f5cf0120fc89852a776828c638fde5fecf75c9e76c"} Oct 30 22:15:02.093544 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.093436 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerStarted","Data":"2c274ba4e270e53a1737f578338758d9a3c9660df9e6e24c104e3e3be2587748"} Oct 30 22:15:02.095182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.095156 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerStarted","Data":"b6519747a92c237e261f90e832524fda23ddda91cfe33f4e513e58d0f2642194"} Oct 30 22:15:02.095267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.095197 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerStarted","Data":"9e8f4acdedc0731523db10762a78000e1e837af8b677befa4f9a9e8ff2650bd5"} Oct 30 22:15:02.096857 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.096838 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" Oct 30 22:15:02.096857 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.096848 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm" event={"ID":"a0bc70b4-aaa3-482e-bd60-08ec61c2e18c","Type":"ContainerDied","Data":"58d204c9a442567cf5015da26def7549fdb0c8b9413bf8d33f0e4471563e84f3"} Oct 30 22:15:02.097040 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.096883 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d204c9a442567cf5015da26def7549fdb0c8b9413bf8d33f0e4471563e84f3" Oct 30 22:15:02.098782 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.098760 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerStarted","Data":"a10bbc1e249654a30d7e2ff0d731b0522d7261a9405b8f2334c1d16a3e0a8fb1"} Oct 30 22:15:02.098887 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.098808 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerStarted","Data":"db592af9ec70697547e28860e8e2a95b705d742f0ec4a5a669eb2ed445825db5"} Oct 30 22:15:02.100804 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.100707 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerStarted","Data":"bc0fce2a44f67866fbe7ffea4d322e70bc0144cd92b1e8e022fef54412197fd1"} Oct 30 22:15:02.100804 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.100765 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerStarted","Data":"912450d3001dff5a133609280778fdc66b99dd112738ed982864ccad1068ffa4"} Oct 30 22:15:02.102565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.102543 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerStarted","Data":"de158a5de01065ac9185ede255dea00e9eceeefcc6692c1fc285b9c66f667416"} Oct 30 22:15:02.102676 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.102584 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerStarted","Data":"081ceff5a85c88b191fc5e235f47ac1e4605dae6bf451d58dba4289588e6c86c"} Oct 30 22:15:02.104461 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.104442 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" event={"ID":"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72","Type":"ContainerStarted","Data":"252109f3bdac550b12fbe401c33dbd0405fd4b9853b5e8a594b4bf5daae05c62"} Oct 30 22:15:02.104577 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.104477 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" event={"ID":"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72","Type":"ContainerStarted","Data":"b9018bb34b07ca7a7b45c69cba7b9650ba64fbce9eb4464df8260faf25ce7a5f"} Oct 30 22:15:02.203471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.203420 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" podStartSLOduration=2.203407575 podStartE2EDuration="2.203407575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.201428192 +0000 UTC m=+1950.882578086" watchObservedRunningTime="2026-10-30 22:15:02.203407575 +0000 UTC m=+1950.884557468" Oct 30 22:15:02.437731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.437045 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" podStartSLOduration=2.437026714 podStartE2EDuration="2.437026714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.434476213 +0000 UTC m=+1951.115626107" watchObservedRunningTime="2026-10-30 22:15:02.437026714 +0000 UTC m=+1951.118176608" Oct 30 22:15:02.551020 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.550973 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" podStartSLOduration=2.550960751 podStartE2EDuration="2.550960751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.550262685 +0000 UTC m=+1951.231412579" watchObservedRunningTime="2026-10-30 22:15:02.550960751 +0000 UTC m=+1951.232110675" Oct 30 22:15:02.721539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.721477 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" podStartSLOduration=2.7214640450000003 podStartE2EDuration="2.721464045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.718026372 +0000 UTC m=+1951.399176266" watchObservedRunningTime="2026-10-30 22:15:02.721464045 +0000 UTC m=+1951.402613963" Oct 30 22:15:02.833213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:02.833167 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" podStartSLOduration=2.8331532470000003 podStartE2EDuration="2.833153247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.832601076 +0000 UTC m=+1951.513750972" watchObservedRunningTime="2026-10-30 22:15:02.833153247 +0000 UTC m=+1951.514303139" Oct 30 22:15:03.065246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.065193 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" podStartSLOduration=3.065179818 podStartE2EDuration="3.065179818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.063296109 +0000 UTC m=+1951.744446000" watchObservedRunningTime="2026-10-30 22:15:03.065179818 +0000 UTC m=+1951.746329711" Oct 30 22:15:03.068390 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.068358 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" podStartSLOduration=3.068345607 podStartE2EDuration="3.068345607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.949549031 +0000 UTC m=+1951.630698925" watchObservedRunningTime="2026-10-30 22:15:03.068345607 +0000 UTC m=+1951.749495497" Oct 30 22:15:03.115639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.115562 4933 generic.go:356] "Generic (PLEG): container finished" podID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerID="de158a5de01065ac9185ede255dea00e9eceeefcc6692c1fc285b9c66f667416" exitCode=60 Oct 30 22:15:03.116177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.115647 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerDied","Data":"de158a5de01065ac9185ede255dea00e9eceeefcc6692c1fc285b9c66f667416"} Oct 30 22:15:03.116177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.115917 4933 scope.go:122] "RemoveContainer" containerID="de158a5de01065ac9185ede255dea00e9eceeefcc6692c1fc285b9c66f667416" Oct 30 22:15:03.117459 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.117433 4933 generic.go:356] "Generic (PLEG): container finished" podID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerID="252109f3bdac550b12fbe401c33dbd0405fd4b9853b5e8a594b4bf5daae05c62" exitCode=60 Oct 30 22:15:03.117582 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.117468 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" event={"ID":"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72","Type":"ContainerDied","Data":"252109f3bdac550b12fbe401c33dbd0405fd4b9853b5e8a594b4bf5daae05c62"} Oct 30 22:15:03.117956 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.117939 4933 scope.go:122] "RemoveContainer" containerID="252109f3bdac550b12fbe401c33dbd0405fd4b9853b5e8a594b4bf5daae05c62" Oct 30 22:15:03.119580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.119558 4933 generic.go:356] "Generic (PLEG): container finished" podID="031c1f3f-7e42-488c-ba8c-36342be21470" containerID="a056f603ebbb945a85920f26860ce9b686e7fa91b22b4669b88bd83388b37e0e" exitCode=60 Oct 30 22:15:03.119677 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.119656 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerDied","Data":"a056f603ebbb945a85920f26860ce9b686e7fa91b22b4669b88bd83388b37e0e"} Oct 30 22:15:03.119957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.119940 4933 scope.go:122] "RemoveContainer" containerID="a056f603ebbb945a85920f26860ce9b686e7fa91b22b4669b88bd83388b37e0e" Oct 30 22:15:03.124538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.124481 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" event={"ID":"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f","Type":"ContainerStarted","Data":"00f30010d6712f65415886a6025ec70f2159974745e828c032186715643eda2e"} Oct 30 22:15:03.126232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.126209 4933 generic.go:356] "Generic (PLEG): container finished" podID="e31c4d05-a234-4df5-a548-0574613c51f6" containerID="8aa3be71ac92f2f6334d1c79d5fc692673b33376b4f83a9cebd6659ce8f8fa98" exitCode=60 Oct 30 22:15:03.126352 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.126309 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerDied","Data":"8aa3be71ac92f2f6334d1c79d5fc692673b33376b4f83a9cebd6659ce8f8fa98"} Oct 30 22:15:03.126911 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.126892 4933 scope.go:122] "RemoveContainer" containerID="8aa3be71ac92f2f6334d1c79d5fc692673b33376b4f83a9cebd6659ce8f8fa98" Oct 30 22:15:03.128191 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.128168 4933 generic.go:356] "Generic (PLEG): container finished" podID="f8d83313-5765-48db-9683-c852af36ad58" containerID="f28311e04131359368ed10f5cf0120fc89852a776828c638fde5fecf75c9e76c" exitCode=60 Oct 30 22:15:03.128398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.128378 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerDied","Data":"f28311e04131359368ed10f5cf0120fc89852a776828c638fde5fecf75c9e76c"} Oct 30 22:15:03.128587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.128566 4933 scope.go:122] "RemoveContainer" containerID="f28311e04131359368ed10f5cf0120fc89852a776828c638fde5fecf75c9e76c" Oct 30 22:15:03.130017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.129996 4933 generic.go:356] "Generic (PLEG): container finished" podID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerID="b6519747a92c237e261f90e832524fda23ddda91cfe33f4e513e58d0f2642194" exitCode=60 Oct 30 22:15:03.130133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.130115 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerDied","Data":"b6519747a92c237e261f90e832524fda23ddda91cfe33f4e513e58d0f2642194"} Oct 30 22:15:03.130381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.130365 4933 scope.go:122] "RemoveContainer" containerID="b6519747a92c237e261f90e832524fda23ddda91cfe33f4e513e58d0f2642194" Oct 30 22:15:03.138791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.138768 4933 generic.go:356] "Generic (PLEG): container finished" podID="1327a728-8c81-4c47-9985-397ff7342c4b" containerID="a10bbc1e249654a30d7e2ff0d731b0522d7261a9405b8f2334c1d16a3e0a8fb1" exitCode=60 Oct 30 22:15:03.138896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.138854 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerDied","Data":"a10bbc1e249654a30d7e2ff0d731b0522d7261a9405b8f2334c1d16a3e0a8fb1"} Oct 30 22:15:03.139075 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.139053 4933 scope.go:122] "RemoveContainer" containerID="a10bbc1e249654a30d7e2ff0d731b0522d7261a9405b8f2334c1d16a3e0a8fb1" Oct 30 22:15:03.171861 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.171827 4933 generic.go:356] "Generic (PLEG): container finished" podID="0408ead9-3525-4ef7-a92f-17189843b02f" containerID="bc0fce2a44f67866fbe7ffea4d322e70bc0144cd92b1e8e022fef54412197fd1" exitCode=60 Oct 30 22:15:03.171997 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.171871 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerDied","Data":"bc0fce2a44f67866fbe7ffea4d322e70bc0144cd92b1e8e022fef54412197fd1"} Oct 30 22:15:03.172394 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.172360 4933 scope.go:122] "RemoveContainer" containerID="bc0fce2a44f67866fbe7ffea4d322e70bc0144cd92b1e8e022fef54412197fd1" Oct 30 22:15:03.182053 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.182009 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" podStartSLOduration=3.181997477 podStartE2EDuration="3.181997477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.181631189 +0000 UTC m=+1951.862781081" watchObservedRunningTime="2026-10-30 22:15:03.181997477 +0000 UTC m=+1951.863147416" Oct 30 22:15:03.871423 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:03.871380 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" podStartSLOduration=3.871368372 podStartE2EDuration="3.871368372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.87056445 +0000 UTC m=+1952.551714344" watchObservedRunningTime="2026-10-30 22:15:03.871368372 +0000 UTC m=+1952.552518296" Oct 30 22:15:04.182340 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.182305 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerStarted","Data":"74d4457a35dbb172d44abb60161cb608c747879b6801e810eeeb44b97de5e041"} Oct 30 22:15:04.184371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.184339 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerStarted","Data":"d8ca0e71a9f516633bda157c08a0123f1550c6f9a637b60961c0bc367c7242fc"} Oct 30 22:15:04.186453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.186429 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerStarted","Data":"cb5d5c8a995b620a191928d6117cee5f846db9eb5db5621bc43494613c46aca5"} Oct 30 22:15:04.188448 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.188421 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" event={"ID":"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72","Type":"ContainerStarted","Data":"54d200cb0b4cc803a0255c3811d02b57af794020a38de2cedb6e8f1187c9c260"} Oct 30 22:15:04.190444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.190419 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerStarted","Data":"cd0a375d7b87d36530a774b096aa5ce7ebfb9991c8a2dc445a5dffc5e96444e2"} Oct 30 22:15:04.192484 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.192461 4933 generic.go:356] "Generic (PLEG): container finished" podID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerID="00f30010d6712f65415886a6025ec70f2159974745e828c032186715643eda2e" exitCode=60 Oct 30 22:15:04.192628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.192518 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" event={"ID":"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f","Type":"ContainerDied","Data":"00f30010d6712f65415886a6025ec70f2159974745e828c032186715643eda2e"} Oct 30 22:15:04.192628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.192566 4933 scope.go:122] "RemoveContainer" containerID="69a4b77de1b721a8c22d001635243ee5975bf1b09892eb7003251adbb49258a8" Oct 30 22:15:04.192782 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.192765 4933 scope.go:122] "RemoveContainer" containerID="00f30010d6712f65415886a6025ec70f2159974745e828c032186715643eda2e" Oct 30 22:15:04.193036 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:04.193011 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq_odh-ai-gateway-infra(0d00942b-1e9e-4b7f-8ca7-91b88e153a8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" Oct 30 22:15:04.194706 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.194681 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerStarted","Data":"81abde7140a22f4023b6c58a3bd82afa606e7a79f2b84a304a600352b4356087"} Oct 30 22:15:04.197386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.197363 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerStarted","Data":"aa26b0584b30f2f54a01ddbf4f1c03dd887f1160c1d5a8d6e2b2155e0a47c269"} Oct 30 22:15:04.199358 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:04.199337 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerStarted","Data":"52a5a94985c2c836dda0b8aba36159331c5e33dd26983d6ea53b005c70c8d99c"} Oct 30 22:15:05.215284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.215252 4933 generic.go:356] "Generic (PLEG): container finished" podID="031c1f3f-7e42-488c-ba8c-36342be21470" containerID="cd0a375d7b87d36530a774b096aa5ce7ebfb9991c8a2dc445a5dffc5e96444e2" exitCode=60 Oct 30 22:15:05.215766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.215330 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerDied","Data":"cd0a375d7b87d36530a774b096aa5ce7ebfb9991c8a2dc445a5dffc5e96444e2"} Oct 30 22:15:05.215766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.215386 4933 scope.go:122] "RemoveContainer" containerID="a056f603ebbb945a85920f26860ce9b686e7fa91b22b4669b88bd83388b37e0e" Oct 30 22:15:05.215766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.215681 4933 scope.go:122] "RemoveContainer" containerID="cd0a375d7b87d36530a774b096aa5ce7ebfb9991c8a2dc445a5dffc5e96444e2" Oct 30 22:15:05.216059 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.216021 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-hghgh_odh-ai-gateway-infra(031c1f3f-7e42-488c-ba8c-36342be21470)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" Oct 30 22:15:05.217659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.217588 4933 scope.go:122] "RemoveContainer" containerID="00f30010d6712f65415886a6025ec70f2159974745e828c032186715643eda2e" Oct 30 22:15:05.217932 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.217910 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq_odh-ai-gateway-infra(0d00942b-1e9e-4b7f-8ca7-91b88e153a8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" Oct 30 22:15:05.219332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.219308 4933 generic.go:356] "Generic (PLEG): container finished" podID="e31c4d05-a234-4df5-a548-0574613c51f6" containerID="81abde7140a22f4023b6c58a3bd82afa606e7a79f2b84a304a600352b4356087" exitCode=60 Oct 30 22:15:05.219440 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.219400 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerDied","Data":"81abde7140a22f4023b6c58a3bd82afa606e7a79f2b84a304a600352b4356087"} Oct 30 22:15:05.219671 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.219655 4933 scope.go:122] "RemoveContainer" containerID="81abde7140a22f4023b6c58a3bd82afa606e7a79f2b84a304a600352b4356087" Oct 30 22:15:05.219956 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.219937 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm_odh-ai-gateway-infra(e31c4d05-a234-4df5-a548-0574613c51f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" Oct 30 22:15:05.221448 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.221430 4933 generic.go:356] "Generic (PLEG): container finished" podID="f8d83313-5765-48db-9683-c852af36ad58" containerID="aa26b0584b30f2f54a01ddbf4f1c03dd887f1160c1d5a8d6e2b2155e0a47c269" exitCode=60 Oct 30 22:15:05.221563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.221509 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerDied","Data":"aa26b0584b30f2f54a01ddbf4f1c03dd887f1160c1d5a8d6e2b2155e0a47c269"} Oct 30 22:15:05.221767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.221750 4933 scope.go:122] "RemoveContainer" containerID="aa26b0584b30f2f54a01ddbf4f1c03dd887f1160c1d5a8d6e2b2155e0a47c269" Oct 30 22:15:05.222038 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.222018 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc_odh-ai-gateway-infra(f8d83313-5765-48db-9683-c852af36ad58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" podUID="f8d83313-5765-48db-9683-c852af36ad58" Oct 30 22:15:05.223522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.223502 4933 generic.go:356] "Generic (PLEG): container finished" podID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerID="52a5a94985c2c836dda0b8aba36159331c5e33dd26983d6ea53b005c70c8d99c" exitCode=60 Oct 30 22:15:05.223628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.223610 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerDied","Data":"52a5a94985c2c836dda0b8aba36159331c5e33dd26983d6ea53b005c70c8d99c"} Oct 30 22:15:05.223831 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.223814 4933 scope.go:122] "RemoveContainer" containerID="52a5a94985c2c836dda0b8aba36159331c5e33dd26983d6ea53b005c70c8d99c" Oct 30 22:15:05.224091 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.224071 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2_odh-ai-gateway-infra(4e7bf9c8-7c39-417b-937e-63516a76f54a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" Oct 30 22:15:05.225824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.225800 4933 generic.go:356] "Generic (PLEG): container finished" podID="1327a728-8c81-4c47-9985-397ff7342c4b" containerID="74d4457a35dbb172d44abb60161cb608c747879b6801e810eeeb44b97de5e041" exitCode=60 Oct 30 22:15:05.225947 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.225933 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerDied","Data":"74d4457a35dbb172d44abb60161cb608c747879b6801e810eeeb44b97de5e041"} Oct 30 22:15:05.226130 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.226112 4933 scope.go:122] "RemoveContainer" containerID="74d4457a35dbb172d44abb60161cb608c747879b6801e810eeeb44b97de5e041" Oct 30 22:15:05.226403 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.226375 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx_odh-ai-gateway-infra(1327a728-8c81-4c47-9985-397ff7342c4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" Oct 30 22:15:05.227967 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.227946 4933 generic.go:356] "Generic (PLEG): container finished" podID="0408ead9-3525-4ef7-a92f-17189843b02f" containerID="d8ca0e71a9f516633bda157c08a0123f1550c6f9a637b60961c0bc367c7242fc" exitCode=60 Oct 30 22:15:05.228055 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.228021 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerDied","Data":"d8ca0e71a9f516633bda157c08a0123f1550c6f9a637b60961c0bc367c7242fc"} Oct 30 22:15:05.228209 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.228195 4933 scope.go:122] "RemoveContainer" containerID="d8ca0e71a9f516633bda157c08a0123f1550c6f9a637b60961c0bc367c7242fc" Oct 30 22:15:05.228452 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.228427 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8_odh-ai-gateway-infra(0408ead9-3525-4ef7-a92f-17189843b02f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" Oct 30 22:15:05.230441 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.230422 4933 scope.go:122] "RemoveContainer" containerID="8aa3be71ac92f2f6334d1c79d5fc692673b33376b4f83a9cebd6659ce8f8fa98" Oct 30 22:15:05.231163 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.231130 4933 generic.go:356] "Generic (PLEG): container finished" podID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerID="cb5d5c8a995b620a191928d6117cee5f846db9eb5db5621bc43494613c46aca5" exitCode=60 Oct 30 22:15:05.231263 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.231206 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerDied","Data":"cb5d5c8a995b620a191928d6117cee5f846db9eb5db5621bc43494613c46aca5"} Oct 30 22:15:05.231408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.231363 4933 scope.go:122] "RemoveContainer" containerID="cb5d5c8a995b620a191928d6117cee5f846db9eb5db5621bc43494613c46aca5" Oct 30 22:15:05.231824 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.231804 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r_odh-ai-gateway-infra(76c59b74-a98f-4464-b7e0-a228677cea1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" Oct 30 22:15:05.234010 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.233987 4933 generic.go:356] "Generic (PLEG): container finished" podID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerID="54d200cb0b4cc803a0255c3811d02b57af794020a38de2cedb6e8f1187c9c260" exitCode=60 Oct 30 22:15:05.234110 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.234065 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" event={"ID":"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72","Type":"ContainerDied","Data":"54d200cb0b4cc803a0255c3811d02b57af794020a38de2cedb6e8f1187c9c260"} Oct 30 22:15:05.234272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.234254 4933 scope.go:122] "RemoveContainer" containerID="54d200cb0b4cc803a0255c3811d02b57af794020a38de2cedb6e8f1187c9c260" Oct 30 22:15:05.234503 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:05.234458 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8_odh-ai-gateway-infra(ea06c18b-0aeb-4d30-a27f-9b08ca70ed72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" Oct 30 22:15:05.244704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.244685 4933 scope.go:122] "RemoveContainer" containerID="f28311e04131359368ed10f5cf0120fc89852a776828c638fde5fecf75c9e76c" Oct 30 22:15:05.258015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.257996 4933 scope.go:122] "RemoveContainer" containerID="b6519747a92c237e261f90e832524fda23ddda91cfe33f4e513e58d0f2642194" Oct 30 22:15:05.272121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.272103 4933 scope.go:122] "RemoveContainer" containerID="a10bbc1e249654a30d7e2ff0d731b0522d7261a9405b8f2334c1d16a3e0a8fb1" Oct 30 22:15:05.287961 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.287942 4933 scope.go:122] "RemoveContainer" containerID="bc0fce2a44f67866fbe7ffea4d322e70bc0144cd92b1e8e022fef54412197fd1" Oct 30 22:15:05.336200 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.336161 4933 scope.go:122] "RemoveContainer" containerID="de158a5de01065ac9185ede255dea00e9eceeefcc6692c1fc285b9c66f667416" Oct 30 22:15:05.370784 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:05.370759 4933 scope.go:122] "RemoveContainer" containerID="252109f3bdac550b12fbe401c33dbd0405fd4b9853b5e8a594b4bf5daae05c62" Oct 30 22:15:06.243479 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.243450 4933 scope.go:122] "RemoveContainer" containerID="cd0a375d7b87d36530a774b096aa5ce7ebfb9991c8a2dc445a5dffc5e96444e2" Oct 30 22:15:06.243926 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.243873 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-hghgh_odh-ai-gateway-infra(031c1f3f-7e42-488c-ba8c-36342be21470)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" Oct 30 22:15:06.245106 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.245081 4933 scope.go:122] "RemoveContainer" containerID="81abde7140a22f4023b6c58a3bd82afa606e7a79f2b84a304a600352b4356087" Oct 30 22:15:06.245377 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.245352 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm_odh-ai-gateway-infra(e31c4d05-a234-4df5-a548-0574613c51f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" Oct 30 22:15:06.246643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.246622 4933 scope.go:122] "RemoveContainer" containerID="aa26b0584b30f2f54a01ddbf4f1c03dd887f1160c1d5a8d6e2b2155e0a47c269" Oct 30 22:15:06.246892 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.246870 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc_odh-ai-gateway-infra(f8d83313-5765-48db-9683-c852af36ad58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" podUID="f8d83313-5765-48db-9683-c852af36ad58" Oct 30 22:15:06.248131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.248113 4933 scope.go:122] "RemoveContainer" containerID="52a5a94985c2c836dda0b8aba36159331c5e33dd26983d6ea53b005c70c8d99c" Oct 30 22:15:06.248393 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.248373 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2_odh-ai-gateway-infra(4e7bf9c8-7c39-417b-937e-63516a76f54a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" Oct 30 22:15:06.249641 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.249622 4933 scope.go:122] "RemoveContainer" containerID="74d4457a35dbb172d44abb60161cb608c747879b6801e810eeeb44b97de5e041" Oct 30 22:15:06.249880 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.249859 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx_odh-ai-gateway-infra(1327a728-8c81-4c47-9985-397ff7342c4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" Oct 30 22:15:06.251084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.251066 4933 scope.go:122] "RemoveContainer" containerID="d8ca0e71a9f516633bda157c08a0123f1550c6f9a637b60961c0bc367c7242fc" Oct 30 22:15:06.251308 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.251288 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8_odh-ai-gateway-infra(0408ead9-3525-4ef7-a92f-17189843b02f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" Oct 30 22:15:06.252735 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.252709 4933 scope.go:122] "RemoveContainer" containerID="cb5d5c8a995b620a191928d6117cee5f846db9eb5db5621bc43494613c46aca5" Oct 30 22:15:06.252961 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.252938 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r_odh-ai-gateway-infra(76c59b74-a98f-4464-b7e0-a228677cea1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" Oct 30 22:15:06.254278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:06.254260 4933 scope.go:122] "RemoveContainer" containerID="54d200cb0b4cc803a0255c3811d02b57af794020a38de2cedb6e8f1187c9c260" Oct 30 22:15:06.254519 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:06.254479 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8_odh-ai-gateway-infra(ea06c18b-0aeb-4d30-a27f-9b08ca70ed72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" Oct 30 22:15:12.742500 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.742448 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk"] Oct 30 22:15:12.744074 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.744049 4933 cpu_manager.go:414] "containerMap: removing container" podUID="a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" containerName="revoke-keys" Oct 30 22:15:12.744246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.744233 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" containerName="revoke-keys" Oct 30 22:15:12.744616 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.744602 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" containerName="revoke-keys" Oct 30 22:15:12.756580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.752426 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.807544 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.807344 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk"] Oct 30 22:15:12.836638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.836602 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.836897 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.836872 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkrz\" (UniqueName: \"kubernetes.io/projected/0fda1b43-5818-458d-b856-49e01cf96286-kube-api-access-tbkrz\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.837185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.837164 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.841677 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.841653 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:12.939402 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.939362 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.939659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.939426 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkrz\" (UniqueName: \"kubernetes.io/projected/0fda1b43-5818-458d-b856-49e01cf96286-kube-api-access-tbkrz\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.939659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.939629 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.940363 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.940322 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.942883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.942840 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:12.998319 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:12.998249 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkrz\" (UniqueName: \"kubernetes.io/projected/0fda1b43-5818-458d-b856-49e01cf96286-kube-api-access-tbkrz\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:13.096694 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:13.096663 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:13.483726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:13.483696 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk"] Oct 30 22:15:13.483910 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:13.483891 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fda1b43_5818_458d_b856_49e01cf96286.slice/crio-edfcd716e9cc98275cc93fc6c5e32ccd8b7a135f6c24e8287b7122b964459cec WatchSource:0}: Error finding container edfcd716e9cc98275cc93fc6c5e32ccd8b7a135f6c24e8287b7122b964459cec: Status 404 returned error can't find the container with id edfcd716e9cc98275cc93fc6c5e32ccd8b7a135f6c24e8287b7122b964459cec Oct 30 22:15:14.344070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:14.344035 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" event={"ID":"0fda1b43-5818-458d-b856-49e01cf96286","Type":"ContainerStarted","Data":"016412c5c9a2d10d28d34e03f456d5ad1dd8f0b2a0d09ff13ef12c44c0d90fcd"} Oct 30 22:15:14.344841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:14.344094 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" event={"ID":"0fda1b43-5818-458d-b856-49e01cf96286","Type":"ContainerStarted","Data":"edfcd716e9cc98275cc93fc6c5e32ccd8b7a135f6c24e8287b7122b964459cec"} Oct 30 22:15:14.459310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:14.459269 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" podStartSLOduration=2.459256146 podStartE2EDuration="2.459256146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:14.45900532 +0000 UTC m=+1963.140155214" watchObservedRunningTime="2026-10-30 22:15:14.459256146 +0000 UTC m=+1963.140406039" Oct 30 22:15:15.355628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:15.355592 4933 generic.go:356] "Generic (PLEG): container finished" podID="0fda1b43-5818-458d-b856-49e01cf96286" containerID="016412c5c9a2d10d28d34e03f456d5ad1dd8f0b2a0d09ff13ef12c44c0d90fcd" exitCode=0 Oct 30 22:15:15.356044 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:15.355628 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" event={"ID":"0fda1b43-5818-458d-b856-49e01cf96286","Type":"ContainerDied","Data":"016412c5c9a2d10d28d34e03f456d5ad1dd8f0b2a0d09ff13ef12c44c0d90fcd"} Oct 30 22:15:15.470564 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:15.470537 4933 scope.go:122] "RemoveContainer" containerID="81abde7140a22f4023b6c58a3bd82afa606e7a79f2b84a304a600352b4356087" Oct 30 22:15:15.470724 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:15.470663 4933 scope.go:122] "RemoveContainer" containerID="00f30010d6712f65415886a6025ec70f2159974745e828c032186715643eda2e" Oct 30 22:15:16.378928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:16.378892 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" event={"ID":"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f","Type":"ContainerStarted","Data":"a315dc6fc13787c9557b2abf1d72426231b61fd353da19b6bba892daeeb38dc4"} Oct 30 22:15:16.381375 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:16.381333 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerStarted","Data":"6d2683c7b6d6a5bb3b7e4b5e6aed51dd2032ace0dc4088bdc901b45cd9c91517"} Oct 30 22:15:16.459096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:16.459070 4933 scope.go:122] "RemoveContainer" containerID="74d4457a35dbb172d44abb60161cb608c747879b6801e810eeeb44b97de5e041" Oct 30 22:15:16.459242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:16.459226 4933 scope.go:122] "RemoveContainer" containerID="d8ca0e71a9f516633bda157c08a0123f1550c6f9a637b60961c0bc367c7242fc" Oct 30 22:15:16.459401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:16.459383 4933 scope.go:122] "RemoveContainer" containerID="cb5d5c8a995b620a191928d6117cee5f846db9eb5db5621bc43494613c46aca5" Oct 30 22:15:16.856970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:16.856944 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:17.001761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.000723 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-xks-ca\") pod \"0fda1b43-5818-458d-b856-49e01cf96286\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " Oct 30 22:15:17.001761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.000800 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fda1b43-5818-458d-b856-49e01cf96286-kube-api-access-tbkrz\" (UniqueName: \"kubernetes.io/projected/0fda1b43-5818-458d-b856-49e01cf96286-kube-api-access-tbkrz\") pod \"0fda1b43-5818-458d-b856-49e01cf96286\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " Oct 30 22:15:17.001761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.001017 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-openshift-ca\") pod \"0fda1b43-5818-458d-b856-49e01cf96286\" (UID: \"0fda1b43-5818-458d-b856-49e01cf96286\") " Oct 30 22:15:17.005523 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.002146 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-openshift-ca" pod "0fda1b43-5818-458d-b856-49e01cf96286" (UID: "0fda1b43-5818-458d-b856-49e01cf96286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:17.034649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.030984 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-xks-ca" pod "0fda1b43-5818-458d-b856-49e01cf96286" (UID: "0fda1b43-5818-458d-b856-49e01cf96286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:17.035948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.035927 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fda1b43-5818-458d-b856-49e01cf96286-kube-api-access-tbkrz" pod "0fda1b43-5818-458d-b856-49e01cf96286" (UID: "0fda1b43-5818-458d-b856-49e01cf96286"). InnerVolumeSpecName "kube-api-access-tbkrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:17.102913 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.102888 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:17.102913 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.102911 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkrz\" (UniqueName: \"kubernetes.io/projected/0fda1b43-5818-458d-b856-49e01cf96286-kube-api-access-tbkrz\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:17.103052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.102923 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fda1b43-5818-458d-b856-49e01cf96286-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:17.390152 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.390122 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" event={"ID":"0fda1b43-5818-458d-b856-49e01cf96286","Type":"ContainerDied","Data":"edfcd716e9cc98275cc93fc6c5e32ccd8b7a135f6c24e8287b7122b964459cec"} Oct 30 22:15:17.390560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.390172 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edfcd716e9cc98275cc93fc6c5e32ccd8b7a135f6c24e8287b7122b964459cec" Oct 30 22:15:17.390560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.390127 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk" Oct 30 22:15:17.392809 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.392783 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerStarted","Data":"896dbddfd57921ca534662139c630694be021d64622d423d5de317444aabb910"} Oct 30 22:15:17.394761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.394740 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerStarted","Data":"03a818d4633169909a71b5059723b49c114f5b594617e85952ea18b2862e5cad"} Oct 30 22:15:17.396830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.396803 4933 generic.go:356] "Generic (PLEG): container finished" podID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerID="a315dc6fc13787c9557b2abf1d72426231b61fd353da19b6bba892daeeb38dc4" exitCode=60 Oct 30 22:15:17.396948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.396873 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" event={"ID":"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f","Type":"ContainerDied","Data":"a315dc6fc13787c9557b2abf1d72426231b61fd353da19b6bba892daeeb38dc4"} Oct 30 22:15:17.396948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.396925 4933 scope.go:122] "RemoveContainer" containerID="00f30010d6712f65415886a6025ec70f2159974745e828c032186715643eda2e" Oct 30 22:15:17.397216 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.397199 4933 scope.go:122] "RemoveContainer" containerID="a315dc6fc13787c9557b2abf1d72426231b61fd353da19b6bba892daeeb38dc4" Oct 30 22:15:17.397545 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:17.397524 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq_odh-ai-gateway-infra(0d00942b-1e9e-4b7f-8ca7-91b88e153a8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" Oct 30 22:15:17.398915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.398897 4933 generic.go:356] "Generic (PLEG): container finished" podID="e31c4d05-a234-4df5-a548-0574613c51f6" containerID="6d2683c7b6d6a5bb3b7e4b5e6aed51dd2032ace0dc4088bdc901b45cd9c91517" exitCode=60 Oct 30 22:15:17.398994 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.398967 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerDied","Data":"6d2683c7b6d6a5bb3b7e4b5e6aed51dd2032ace0dc4088bdc901b45cd9c91517"} Oct 30 22:15:17.399281 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.399265 4933 scope.go:122] "RemoveContainer" containerID="6d2683c7b6d6a5bb3b7e4b5e6aed51dd2032ace0dc4088bdc901b45cd9c91517" Oct 30 22:15:17.399636 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:17.399583 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm_odh-ai-gateway-infra(e31c4d05-a234-4df5-a548-0574613c51f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" Oct 30 22:15:17.401338 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.401318 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerStarted","Data":"1ee4dabb08a25ca7672476c2b77f487af65a5db6ffe901a60839659e229ca41c"} Oct 30 22:15:17.413452 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.413434 4933 scope.go:122] "RemoveContainer" containerID="81abde7140a22f4023b6c58a3bd82afa606e7a79f2b84a304a600352b4356087" Oct 30 22:15:17.459366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.459343 4933 scope.go:122] "RemoveContainer" containerID="54d200cb0b4cc803a0255c3811d02b57af794020a38de2cedb6e8f1187c9c260" Oct 30 22:15:17.510926 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.510905 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq"] Oct 30 22:15:17.628471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:17.628445 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm"] Oct 30 22:15:18.414238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.414207 4933 generic.go:356] "Generic (PLEG): container finished" podID="1327a728-8c81-4c47-9985-397ff7342c4b" containerID="1ee4dabb08a25ca7672476c2b77f487af65a5db6ffe901a60839659e229ca41c" exitCode=60 Oct 30 22:15:18.414643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.414281 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerDied","Data":"1ee4dabb08a25ca7672476c2b77f487af65a5db6ffe901a60839659e229ca41c"} Oct 30 22:15:18.414643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.414388 4933 scope.go:122] "RemoveContainer" containerID="74d4457a35dbb172d44abb60161cb608c747879b6801e810eeeb44b97de5e041" Oct 30 22:15:18.414819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.414803 4933 scope.go:122] "RemoveContainer" containerID="1ee4dabb08a25ca7672476c2b77f487af65a5db6ffe901a60839659e229ca41c" Oct 30 22:15:18.415222 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:18.415197 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx_odh-ai-gateway-infra(1327a728-8c81-4c47-9985-397ff7342c4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" Oct 30 22:15:18.418335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.418310 4933 generic.go:356] "Generic (PLEG): container finished" podID="0408ead9-3525-4ef7-a92f-17189843b02f" containerID="896dbddfd57921ca534662139c630694be021d64622d423d5de317444aabb910" exitCode=60 Oct 30 22:15:18.418465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.418383 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerDied","Data":"896dbddfd57921ca534662139c630694be021d64622d423d5de317444aabb910"} Oct 30 22:15:18.418880 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.418844 4933 scope.go:122] "RemoveContainer" containerID="896dbddfd57921ca534662139c630694be021d64622d423d5de317444aabb910" Oct 30 22:15:18.419407 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:18.419232 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8_odh-ai-gateway-infra(0408ead9-3525-4ef7-a92f-17189843b02f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" Oct 30 22:15:18.421666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.421642 4933 generic.go:356] "Generic (PLEG): container finished" podID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerID="03a818d4633169909a71b5059723b49c114f5b594617e85952ea18b2862e5cad" exitCode=60 Oct 30 22:15:18.421779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.421701 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerDied","Data":"03a818d4633169909a71b5059723b49c114f5b594617e85952ea18b2862e5cad"} Oct 30 22:15:18.422125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.422098 4933 scope.go:122] "RemoveContainer" containerID="03a818d4633169909a71b5059723b49c114f5b594617e85952ea18b2862e5cad" Oct 30 22:15:18.422453 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:18.422429 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r_odh-ai-gateway-infra(76c59b74-a98f-4464-b7e0-a228677cea1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" Oct 30 22:15:18.426579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.424524 4933 generic.go:356] "Generic (PLEG): container finished" podID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerID="cbe11c1cc2da58eae3754b06f7c6a05819cb1293c2bd4fbe948b150b941f129b" exitCode=60 Oct 30 22:15:18.426579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.424600 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" event={"ID":"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72","Type":"ContainerDied","Data":"cbe11c1cc2da58eae3754b06f7c6a05819cb1293c2bd4fbe948b150b941f129b"} Oct 30 22:15:18.426579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.424921 4933 scope.go:122] "RemoveContainer" containerID="cbe11c1cc2da58eae3754b06f7c6a05819cb1293c2bd4fbe948b150b941f129b" Oct 30 22:15:18.426579 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:18.425171 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8_odh-ai-gateway-infra(ea06c18b-0aeb-4d30-a27f-9b08ca70ed72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" Oct 30 22:15:18.430645 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.430618 4933 scope.go:122] "RemoveContainer" containerID="d8ca0e71a9f516633bda157c08a0123f1550c6f9a637b60961c0bc367c7242fc" Oct 30 22:15:18.447808 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.447248 4933 scope.go:122] "RemoveContainer" containerID="cb5d5c8a995b620a191928d6117cee5f846db9eb5db5621bc43494613c46aca5" Oct 30 22:15:18.458647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.458628 4933 scope.go:122] "RemoveContainer" containerID="cd0a375d7b87d36530a774b096aa5ce7ebfb9991c8a2dc445a5dffc5e96444e2" Oct 30 22:15:18.459325 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.459308 4933 scope.go:122] "RemoveContainer" containerID="aa26b0584b30f2f54a01ddbf4f1c03dd887f1160c1d5a8d6e2b2155e0a47c269" Oct 30 22:15:18.470838 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.470299 4933 scope.go:122] "RemoveContainer" containerID="54d200cb0b4cc803a0255c3811d02b57af794020a38de2cedb6e8f1187c9c260" Oct 30 22:15:18.528164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.528141 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8"] Oct 30 22:15:18.621165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.621136 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:18.645904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.645873 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r"] Oct 30 22:15:18.663808 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.663782 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:18.729937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.729833 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31c4d05-a234-4df5-a548-0574613c51f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31c4d05-a234-4df5-a548-0574613c51f6-maas-api-service-ca\") pod \"e31c4d05-a234-4df5-a548-0574613c51f6\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " Oct 30 22:15:18.729937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.729906 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31c4d05-a234-4df5-a548-0574613c51f6-kube-api-access-k54d4\" (UniqueName: \"kubernetes.io/projected/e31c4d05-a234-4df5-a548-0574613c51f6-kube-api-access-k54d4\") pod \"e31c4d05-a234-4df5-a548-0574613c51f6\" (UID: \"e31c4d05-a234-4df5-a548-0574613c51f6\") " Oct 30 22:15:18.737531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.730332 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31c4d05-a234-4df5-a548-0574613c51f6-maas-api-service-ca" pod "e31c4d05-a234-4df5-a548-0574613c51f6" (UID: "e31c4d05-a234-4df5-a548-0574613c51f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.737531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.730761 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e31c4d05-a234-4df5-a548-0574613c51f6-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.737531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.733526 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31c4d05-a234-4df5-a548-0574613c51f6-kube-api-access-k54d4" pod "e31c4d05-a234-4df5-a548-0574613c51f6" (UID: "e31c4d05-a234-4df5-a548-0574613c51f6"). InnerVolumeSpecName "kube-api-access-k54d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.831591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.831478 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-maas-api-service-ca\") pod \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " Oct 30 22:15:18.831591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.831578 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-kube-api-access-7fc4g\" (UniqueName: \"kubernetes.io/projected/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-kube-api-access-7fc4g\") pod \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\" (UID: \"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f\") " Oct 30 22:15:18.831875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.831851 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-maas-api-service-ca" pod "0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" (UID: "0d00942b-1e9e-4b7f-8ca7-91b88e153a8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.832190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.832169 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.832451 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.832194 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k54d4\" (UniqueName: \"kubernetes.io/projected/e31c4d05-a234-4df5-a548-0574613c51f6-kube-api-access-k54d4\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:18.834849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.834809 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-kube-api-access-7fc4g" pod "0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" (UID: "0d00942b-1e9e-4b7f-8ca7-91b88e153a8f"). InnerVolumeSpecName "kube-api-access-7fc4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.933304 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.933278 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx"] Oct 30 22:15:18.933768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:18.933743 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fc4g\" (UniqueName: \"kubernetes.io/projected/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f-kube-api-access-7fc4g\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.435230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.435194 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerStarted","Data":"0bc3a44089c5b9a21d2bc3a980ab34a02197cca4ca4ef56a5bea715b168bfe96"} Oct 30 22:15:19.437157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.437119 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" event={"ID":"0d00942b-1e9e-4b7f-8ca7-91b88e153a8f","Type":"ContainerDied","Data":"00219bf6002d0529807e39df5e1f7fb3d9e7d0ecad04bdcd2ec9f4c46c1179d2"} Oct 30 22:15:19.437296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.437198 4933 scope.go:122] "RemoveContainer" containerID="a315dc6fc13787c9557b2abf1d72426231b61fd353da19b6bba892daeeb38dc4" Oct 30 22:15:19.437659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.437641 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq" Oct 30 22:15:19.440853 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.440828 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" event={"ID":"e31c4d05-a234-4df5-a548-0574613c51f6","Type":"ContainerDied","Data":"bc3d70084d8c15a48476aa0db2a8f1fd6a54eef032b1bacef84235597c28f689"} Oct 30 22:15:19.440968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.440917 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm" Oct 30 22:15:19.455544 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.455520 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerStarted","Data":"0c2c790f7df2483e806eccae9628a14bba80e53e0cae889613772944fd63e8ff"} Oct 30 22:15:19.459462 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.459442 4933 scope.go:122] "RemoveContainer" containerID="52a5a94985c2c836dda0b8aba36159331c5e33dd26983d6ea53b005c70c8d99c" Oct 30 22:15:19.462264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.462244 4933 scope.go:122] "RemoveContainer" containerID="6d2683c7b6d6a5bb3b7e4b5e6aed51dd2032ace0dc4088bdc901b45cd9c91517" Oct 30 22:15:19.787000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.786974 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:19.791227 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.791204 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:19.807636 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.804532 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq"] Oct 30 22:15:19.810505 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.810451 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-0a7679-29889975-cftfq"] Oct 30 22:15:19.816815 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.816796 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:19.849185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849155 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0408ead9-3525-4ef7-a92f-17189843b02f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0408ead9-3525-4ef7-a92f-17189843b02f-maas-api-service-ca\") pod \"0408ead9-3525-4ef7-a92f-17189843b02f\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " Oct 30 22:15:19.849332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849249 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1327a728-8c81-4c47-9985-397ff7342c4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1327a728-8c81-4c47-9985-397ff7342c4b-maas-api-service-ca\") pod \"1327a728-8c81-4c47-9985-397ff7342c4b\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " Oct 30 22:15:19.849332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849284 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c59b74-a98f-4464-b7e0-a228677cea1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c59b74-a98f-4464-b7e0-a228677cea1d-maas-api-service-ca\") pod \"76c59b74-a98f-4464-b7e0-a228677cea1d\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " Oct 30 22:15:19.849446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849357 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c59b74-a98f-4464-b7e0-a228677cea1d-kube-api-access-rvsx9\" (UniqueName: \"kubernetes.io/projected/76c59b74-a98f-4464-b7e0-a228677cea1d-kube-api-access-rvsx9\") pod \"76c59b74-a98f-4464-b7e0-a228677cea1d\" (UID: \"76c59b74-a98f-4464-b7e0-a228677cea1d\") " Oct 30 22:15:19.849545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849448 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0408ead9-3525-4ef7-a92f-17189843b02f-kube-api-access-2ncx9\" (UniqueName: \"kubernetes.io/projected/0408ead9-3525-4ef7-a92f-17189843b02f-kube-api-access-2ncx9\") pod \"0408ead9-3525-4ef7-a92f-17189843b02f\" (UID: \"0408ead9-3525-4ef7-a92f-17189843b02f\") " Oct 30 22:15:19.849616 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849558 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1327a728-8c81-4c47-9985-397ff7342c4b-kube-api-access-mdbxr\" (UniqueName: \"kubernetes.io/projected/1327a728-8c81-4c47-9985-397ff7342c4b-kube-api-access-mdbxr\") pod \"1327a728-8c81-4c47-9985-397ff7342c4b\" (UID: \"1327a728-8c81-4c47-9985-397ff7342c4b\") " Oct 30 22:15:19.849673 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849614 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0408ead9-3525-4ef7-a92f-17189843b02f-maas-api-service-ca" pod "0408ead9-3525-4ef7-a92f-17189843b02f" (UID: "0408ead9-3525-4ef7-a92f-17189843b02f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.849673 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849662 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c59b74-a98f-4464-b7e0-a228677cea1d-maas-api-service-ca" pod "76c59b74-a98f-4464-b7e0-a228677cea1d" (UID: "76c59b74-a98f-4464-b7e0-a228677cea1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.849782 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.849660 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1327a728-8c81-4c47-9985-397ff7342c4b-maas-api-service-ca" pod "1327a728-8c81-4c47-9985-397ff7342c4b" (UID: "1327a728-8c81-4c47-9985-397ff7342c4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:19.850210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.850186 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0408ead9-3525-4ef7-a92f-17189843b02f-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.850309 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.850214 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1327a728-8c81-4c47-9985-397ff7342c4b-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.850309 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.850229 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c59b74-a98f-4464-b7e0-a228677cea1d-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.852567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.852543 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1327a728-8c81-4c47-9985-397ff7342c4b-kube-api-access-mdbxr" pod "1327a728-8c81-4c47-9985-397ff7342c4b" (UID: "1327a728-8c81-4c47-9985-397ff7342c4b"). InnerVolumeSpecName "kube-api-access-mdbxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.852710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.852559 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0408ead9-3525-4ef7-a92f-17189843b02f-kube-api-access-2ncx9" pod "0408ead9-3525-4ef7-a92f-17189843b02f" (UID: "0408ead9-3525-4ef7-a92f-17189843b02f"). InnerVolumeSpecName "kube-api-access-2ncx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.852870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.852848 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c59b74-a98f-4464-b7e0-a228677cea1d-kube-api-access-rvsx9" pod "76c59b74-a98f-4464-b7e0-a228677cea1d" (UID: "76c59b74-a98f-4464-b7e0-a228677cea1d"). InnerVolumeSpecName "kube-api-access-rvsx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:19.901799 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.901759 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8"] Oct 30 22:15:19.979293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.979045 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ncx9\" (UniqueName: \"kubernetes.io/projected/0408ead9-3525-4ef7-a92f-17189843b02f-kube-api-access-2ncx9\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.979293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.979080 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdbxr\" (UniqueName: \"kubernetes.io/projected/1327a728-8c81-4c47-9985-397ff7342c4b-kube-api-access-mdbxr\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:19.979293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:19.979092 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvsx9\" (UniqueName: \"kubernetes.io/projected/76c59b74-a98f-4464-b7e0-a228677cea1d-kube-api-access-rvsx9\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.130349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.130320 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:20.180813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.180778 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-kube-api-access-6zjwb\" (UniqueName: \"kubernetes.io/projected/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-kube-api-access-6zjwb\") pod \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " Oct 30 22:15:20.181045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.181027 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-maas-api-service-ca\") pod \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\" (UID: \"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72\") " Oct 30 22:15:20.181390 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.181365 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-maas-api-service-ca" pod "ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" (UID: "ea06c18b-0aeb-4d30-a27f-9b08ca70ed72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.181728 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.181708 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.183073 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.183053 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-kube-api-access-6zjwb" pod "ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" (UID: "ea06c18b-0aeb-4d30-a27f-9b08ca70ed72"). InnerVolumeSpecName "kube-api-access-6zjwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.276981 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.276948 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm"] Oct 30 22:15:20.282135 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.282105 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-e431fc-29889975-p49vm"] Oct 30 22:15:20.283121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.283096 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zjwb\" (UniqueName: \"kubernetes.io/projected/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72-kube-api-access-6zjwb\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.488052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.488024 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" Oct 30 22:15:20.488052 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.488039 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8" event={"ID":"0408ead9-3525-4ef7-a92f-17189843b02f","Type":"ContainerDied","Data":"912450d3001dff5a133609280778fdc66b99dd112738ed982864ccad1068ffa4"} Oct 30 22:15:20.488659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.488106 4933 scope.go:122] "RemoveContainer" containerID="896dbddfd57921ca534662139c630694be021d64622d423d5de317444aabb910" Oct 30 22:15:20.490611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.490582 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" Oct 30 22:15:20.490752 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.490590 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r" event={"ID":"76c59b74-a98f-4464-b7e0-a228677cea1d","Type":"ContainerDied","Data":"081ceff5a85c88b191fc5e235f47ac1e4605dae6bf451d58dba4289588e6c86c"} Oct 30 22:15:20.492536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.492506 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" Oct 30 22:15:20.492536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.492528 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8" event={"ID":"ea06c18b-0aeb-4d30-a27f-9b08ca70ed72","Type":"ContainerDied","Data":"b9018bb34b07ca7a7b45c69cba7b9650ba64fbce9eb4464df8260faf25ce7a5f"} Oct 30 22:15:20.500098 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.500070 4933 generic.go:356] "Generic (PLEG): container finished" podID="031c1f3f-7e42-488c-ba8c-36342be21470" containerID="0bc3a44089c5b9a21d2bc3a980ab34a02197cca4ca4ef56a5bea715b168bfe96" exitCode=60 Oct 30 22:15:20.500218 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.500145 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerDied","Data":"0bc3a44089c5b9a21d2bc3a980ab34a02197cca4ca4ef56a5bea715b168bfe96"} Oct 30 22:15:20.500926 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.500681 4933 scope.go:122] "RemoveContainer" containerID="0bc3a44089c5b9a21d2bc3a980ab34a02197cca4ca4ef56a5bea715b168bfe96" Oct 30 22:15:20.501031 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:20.500994 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-hghgh_odh-ai-gateway-infra(031c1f3f-7e42-488c-ba8c-36342be21470)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" Oct 30 22:15:20.504177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.504155 4933 scope.go:122] "RemoveContainer" containerID="03a818d4633169909a71b5059723b49c114f5b594617e85952ea18b2862e5cad" Oct 30 22:15:20.504671 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.504646 4933 generic.go:356] "Generic (PLEG): container finished" podID="f8d83313-5765-48db-9683-c852af36ad58" containerID="0c2c790f7df2483e806eccae9628a14bba80e53e0cae889613772944fd63e8ff" exitCode=60 Oct 30 22:15:20.504780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.504667 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerDied","Data":"0c2c790f7df2483e806eccae9628a14bba80e53e0cae889613772944fd63e8ff"} Oct 30 22:15:20.505082 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.505039 4933 scope.go:122] "RemoveContainer" containerID="0c2c790f7df2483e806eccae9628a14bba80e53e0cae889613772944fd63e8ff" Oct 30 22:15:20.505382 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:20.505360 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc_odh-ai-gateway-infra(f8d83313-5765-48db-9683-c852af36ad58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" podUID="f8d83313-5765-48db-9683-c852af36ad58" Oct 30 22:15:20.506974 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.506952 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerStarted","Data":"1a9a028abed47e3939587d945d0a5ad5c6758a205cdb41058efaa6b57b6d71ab"} Oct 30 22:15:20.508435 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.508418 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" event={"ID":"1327a728-8c81-4c47-9985-397ff7342c4b","Type":"ContainerDied","Data":"db592af9ec70697547e28860e8e2a95b705d742f0ec4a5a669eb2ed445825db5"} Oct 30 22:15:20.508540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.508439 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx" Oct 30 22:15:20.561200 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.561171 4933 scope.go:122] "RemoveContainer" containerID="cbe11c1cc2da58eae3754b06f7c6a05819cb1293c2bd4fbe948b150b941f129b" Oct 30 22:15:20.570936 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.570907 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh"] Oct 30 22:15:20.599131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.599113 4933 scope.go:122] "RemoveContainer" containerID="cd0a375d7b87d36530a774b096aa5ce7ebfb9991c8a2dc445a5dffc5e96444e2" Oct 30 22:15:20.613016 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.612993 4933 scope.go:122] "RemoveContainer" containerID="aa26b0584b30f2f54a01ddbf4f1c03dd887f1160c1d5a8d6e2b2155e0a47c269" Oct 30 22:15:20.626775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.626756 4933 scope.go:122] "RemoveContainer" containerID="1ee4dabb08a25ca7672476c2b77f487af65a5db6ffe901a60839659e229ca41c" Oct 30 22:15:20.866727 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.866654 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx"] Oct 30 22:15:20.874698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:20.874669 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5d36f6-29889975-hxmpx"] Oct 30 22:15:21.056664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.056633 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc"] Oct 30 22:15:21.096619 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.096585 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8"] Oct 30 22:15:21.101077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.101050 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7c826c-29889975-wttb8"] Oct 30 22:15:21.124448 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.124377 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:15:21.448625 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.448548 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r"] Oct 30 22:15:21.455172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.455145 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9f8aea-29889975-8kf8r"] Oct 30 22:15:21.475721 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.475691 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" path="/var/lib/kubelet/pods/0408ead9-3525-4ef7-a92f-17189843b02f/volumes" Oct 30 22:15:21.476160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.476146 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" path="/var/lib/kubelet/pods/0d00942b-1e9e-4b7f-8ca7-91b88e153a8f/volumes" Oct 30 22:15:21.476612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.476598 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" path="/var/lib/kubelet/pods/1327a728-8c81-4c47-9985-397ff7342c4b/volumes" Oct 30 22:15:21.477309 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.477294 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" path="/var/lib/kubelet/pods/76c59b74-a98f-4464-b7e0-a228677cea1d/volumes" Oct 30 22:15:21.477736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.477723 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" path="/var/lib/kubelet/pods/e31c4d05-a234-4df5-a548-0574613c51f6/volumes" Oct 30 22:15:21.523719 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.523696 4933 generic.go:356] "Generic (PLEG): container finished" podID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerID="1a9a028abed47e3939587d945d0a5ad5c6758a205cdb41058efaa6b57b6d71ab" exitCode=60 Oct 30 22:15:21.524092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.523759 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerDied","Data":"1a9a028abed47e3939587d945d0a5ad5c6758a205cdb41058efaa6b57b6d71ab"} Oct 30 22:15:21.524092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.523812 4933 scope.go:122] "RemoveContainer" containerID="52a5a94985c2c836dda0b8aba36159331c5e33dd26983d6ea53b005c70c8d99c" Oct 30 22:15:21.524198 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.524142 4933 scope.go:122] "RemoveContainer" containerID="1a9a028abed47e3939587d945d0a5ad5c6758a205cdb41058efaa6b57b6d71ab" Oct 30 22:15:21.524450 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:21.524426 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2_odh-ai-gateway-infra(4e7bf9c8-7c39-417b-937e-63516a76f54a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" Oct 30 22:15:21.563190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.563142 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8"] Oct 30 22:15:21.569395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.569370 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d516c4-29889975-z4ct8"] Oct 30 22:15:21.700292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.700223 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:21.763870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.763840 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:21.822565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.814847 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031c1f3f-7e42-488c-ba8c-36342be21470-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c1f3f-7e42-488c-ba8c-36342be21470-maas-api-service-ca\") pod \"031c1f3f-7e42-488c-ba8c-36342be21470\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " Oct 30 22:15:21.822565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.814999 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031c1f3f-7e42-488c-ba8c-36342be21470-kube-api-access-mhfnk\" (UniqueName: \"kubernetes.io/projected/031c1f3f-7e42-488c-ba8c-36342be21470-kube-api-access-mhfnk\") pod \"031c1f3f-7e42-488c-ba8c-36342be21470\" (UID: \"031c1f3f-7e42-488c-ba8c-36342be21470\") " Oct 30 22:15:21.822565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.815035 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d83313-5765-48db-9683-c852af36ad58-kube-api-access-vzglx\" (UniqueName: \"kubernetes.io/projected/f8d83313-5765-48db-9683-c852af36ad58-kube-api-access-vzglx\") pod \"f8d83313-5765-48db-9683-c852af36ad58\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " Oct 30 22:15:21.822565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.815100 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d83313-5765-48db-9683-c852af36ad58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d83313-5765-48db-9683-c852af36ad58-maas-api-service-ca\") pod \"f8d83313-5765-48db-9683-c852af36ad58\" (UID: \"f8d83313-5765-48db-9683-c852af36ad58\") " Oct 30 22:15:21.822565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.815235 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031c1f3f-7e42-488c-ba8c-36342be21470-maas-api-service-ca" pod "031c1f3f-7e42-488c-ba8c-36342be21470" (UID: "031c1f3f-7e42-488c-ba8c-36342be21470"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.822565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.816755 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d83313-5765-48db-9683-c852af36ad58-maas-api-service-ca" pod "f8d83313-5765-48db-9683-c852af36ad58" (UID: "f8d83313-5765-48db-9683-c852af36ad58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.822565 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.820455 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d83313-5765-48db-9683-c852af36ad58-kube-api-access-vzglx" pod "f8d83313-5765-48db-9683-c852af36ad58" (UID: "f8d83313-5765-48db-9683-c852af36ad58"). InnerVolumeSpecName "kube-api-access-vzglx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.827516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.824917 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzglx\" (UniqueName: \"kubernetes.io/projected/f8d83313-5765-48db-9683-c852af36ad58-kube-api-access-vzglx\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.827516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.824948 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d83313-5765-48db-9683-c852af36ad58-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.827516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.824961 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c1f3f-7e42-488c-ba8c-36342be21470-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.840859 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.837219 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031c1f3f-7e42-488c-ba8c-36342be21470-kube-api-access-mhfnk" pod "031c1f3f-7e42-488c-ba8c-36342be21470" (UID: "031c1f3f-7e42-488c-ba8c-36342be21470"). InnerVolumeSpecName "kube-api-access-mhfnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.926634 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:21.926608 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhfnk\" (UniqueName: \"kubernetes.io/projected/031c1f3f-7e42-488c-ba8c-36342be21470-kube-api-access-mhfnk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.350211 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.350178 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2"] Oct 30 22:15:22.535259 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.535234 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" Oct 30 22:15:22.535792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.535260 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh" event={"ID":"031c1f3f-7e42-488c-ba8c-36342be21470","Type":"ContainerDied","Data":"f5a766a05c86c2c847fea4df99dccaf6d3f56f9abe76879e0c650b843a0ae958"} Oct 30 22:15:22.535792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.535347 4933 scope.go:122] "RemoveContainer" containerID="0bc3a44089c5b9a21d2bc3a980ab34a02197cca4ca4ef56a5bea715b168bfe96" Oct 30 22:15:22.536970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.536945 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" event={"ID":"f8d83313-5765-48db-9683-c852af36ad58","Type":"ContainerDied","Data":"2c274ba4e270e53a1737f578338758d9a3c9660df9e6e24c104e3e3be2587748"} Oct 30 22:15:22.537089 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.536987 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc" Oct 30 22:15:22.549331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.549313 4933 scope.go:122] "RemoveContainer" containerID="0c2c790f7df2483e806eccae9628a14bba80e53e0cae889613772944fd63e8ff" Oct 30 22:15:22.705637 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.705611 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh"] Oct 30 22:15:22.709517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.709485 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-hghgh"] Oct 30 22:15:22.711142 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.711118 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:22.819419 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.819387 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc"] Oct 30 22:15:22.823056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.823031 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f1d78db-29889975-5n7hc"] Oct 30 22:15:22.837813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.837791 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7bf9c8-7c39-417b-937e-63516a76f54a-kube-api-access-8ljpw\" (UniqueName: \"kubernetes.io/projected/4e7bf9c8-7c39-417b-937e-63516a76f54a-kube-api-access-8ljpw\") pod \"4e7bf9c8-7c39-417b-937e-63516a76f54a\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " Oct 30 22:15:22.837916 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.837833 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7bf9c8-7c39-417b-937e-63516a76f54a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7bf9c8-7c39-417b-937e-63516a76f54a-maas-api-service-ca\") pod \"4e7bf9c8-7c39-417b-937e-63516a76f54a\" (UID: \"4e7bf9c8-7c39-417b-937e-63516a76f54a\") " Oct 30 22:15:22.838209 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.838189 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7bf9c8-7c39-417b-937e-63516a76f54a-maas-api-service-ca" pod "4e7bf9c8-7c39-417b-937e-63516a76f54a" (UID: "4e7bf9c8-7c39-417b-937e-63516a76f54a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:22.840099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.840076 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7bf9c8-7c39-417b-937e-63516a76f54a-kube-api-access-8ljpw" pod "4e7bf9c8-7c39-417b-937e-63516a76f54a" (UID: "4e7bf9c8-7c39-417b-937e-63516a76f54a"). InnerVolumeSpecName "kube-api-access-8ljpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:22.939662 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.939610 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljpw\" (UniqueName: \"kubernetes.io/projected/4e7bf9c8-7c39-417b-937e-63516a76f54a-kube-api-access-8ljpw\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.939662 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:22.939632 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e7bf9c8-7c39-417b-937e-63516a76f54a-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:23.468288 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.468258 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" path="/var/lib/kubelet/pods/031c1f3f-7e42-488c-ba8c-36342be21470/volumes" Oct 30 22:15:23.468716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.468702 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" path="/var/lib/kubelet/pods/ea06c18b-0aeb-4d30-a27f-9b08ca70ed72/volumes" Oct 30 22:15:23.469084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.469071 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d83313-5765-48db-9683-c852af36ad58" path="/var/lib/kubelet/pods/f8d83313-5765-48db-9683-c852af36ad58/volumes" Oct 30 22:15:23.551594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.551567 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" Oct 30 22:15:23.552078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.551602 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2" event={"ID":"4e7bf9c8-7c39-417b-937e-63516a76f54a","Type":"ContainerDied","Data":"9e8f4acdedc0731523db10762a78000e1e837af8b677befa4f9a9e8ff2650bd5"} Oct 30 22:15:23.552078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.551676 4933 scope.go:122] "RemoveContainer" containerID="1a9a028abed47e3939587d945d0a5ad5c6758a205cdb41058efaa6b57b6d71ab" Oct 30 22:15:23.719619 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.719542 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2"] Oct 30 22:15:23.769713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:23.769685 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0f42cb-29889975-kc7q2"] Oct 30 22:15:25.468105 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:25.468061 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" path="/var/lib/kubelet/pods/4e7bf9c8-7c39-417b-937e-63516a76f54a/volumes" Oct 30 22:15:28.545256 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:28.545222 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:15:29.137730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.137694 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf"] Oct 30 22:15:29.138532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138503 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.138532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138526 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.138532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138539 4933 cpu_manager.go:414] "containerMap: removing container" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138544 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138551 4933 cpu_manager.go:414] "containerMap: removing container" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138556 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138581 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138589 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138600 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138604 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138612 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138616 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138623 4933 cpu_manager.go:414] "containerMap: removing container" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138628 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138633 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138638 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138654 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138661 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138677 4933 cpu_manager.go:414] "containerMap: removing container" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138683 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138697 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138703 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138716 4933 cpu_manager.go:414] "containerMap: removing container" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138722 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138739 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138749 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138762 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138769 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138780 4933 cpu_manager.go:414] "containerMap: removing container" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:29.138778 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138788 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138799 4933 cpu_manager.go:414] "containerMap: removing container" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138806 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138821 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138828 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138838 4933 cpu_manager.go:414] "containerMap: removing container" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138844 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138854 4933 cpu_manager.go:414] "containerMap: removing container" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138861 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138872 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0fda1b43-5818-458d-b856-49e01cf96286" containerName="revoke-keys" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138879 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fda1b43-5818-458d-b856-49e01cf96286" containerName="revoke-keys" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138903 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138910 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138929 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138936 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138954 4933 cpu_manager.go:414] "containerMap: removing container" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.138960 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139122 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139133 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139146 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139158 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139166 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139175 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fda1b43-5818-458d-b856-49e01cf96286" containerName="revoke-keys" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139184 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139192 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139202 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139211 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139220 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7bf9c8-7c39-417b-937e-63516a76f54a" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139229 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139237 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139246 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139253 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139263 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139271 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139278 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139287 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139296 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139303 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:29.139768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.139311 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea06c18b-0aeb-4d30-a27f-9b08ca70ed72" containerName="cleanup" Oct 30 22:15:29.144239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.144221 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.157524 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.157482 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.157643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.157594 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tqrh\" (UniqueName: \"kubernetes.io/projected/bf216553-f6e3-407d-9f44-2e2fd786f836-kube-api-access-5tqrh\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.157774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.157740 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.198732 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.198706 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf"] Oct 30 22:15:29.244063 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.244031 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:15:29.244063 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.244049 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:29.244244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.244075 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:29.248185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.248164 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:29.258855 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.258833 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.258950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.258883 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tqrh\" (UniqueName: \"kubernetes.io/projected/bf216553-f6e3-407d-9f44-2e2fd786f836-kube-api-access-5tqrh\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.259012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.259003 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.259482 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.259462 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.261547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.261517 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.316762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.316735 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tqrh\" (UniqueName: \"kubernetes.io/projected/bf216553-f6e3-407d-9f44-2e2fd786f836-kube-api-access-5tqrh\") pod \"maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.459271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.459197 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:29.841269 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:29.841243 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf216553_f6e3_407d_9f44_2e2fd786f836.slice/crio-c8209b051388ce3f0f46f62f70df13e149bb072d1b6b7679796cd0764b06fe9a WatchSource:0}: Error finding container c8209b051388ce3f0f46f62f70df13e149bb072d1b6b7679796cd0764b06fe9a: Status 404 returned error can't find the container with id c8209b051388ce3f0f46f62f70df13e149bb072d1b6b7679796cd0764b06fe9a Oct 30 22:15:29.841788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:29.841744 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf"] Oct 30 22:15:30.617007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:30.616974 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" event={"ID":"bf216553-f6e3-407d-9f44-2e2fd786f836","Type":"ContainerStarted","Data":"456f3c0597c47b86fd25ce5c16095b7a16f6d893211a4be409262c2ae92473cd"} Oct 30 22:15:30.617201 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:30.617029 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" event={"ID":"bf216553-f6e3-407d-9f44-2e2fd786f836","Type":"ContainerStarted","Data":"c8209b051388ce3f0f46f62f70df13e149bb072d1b6b7679796cd0764b06fe9a"} Oct 30 22:15:30.735352 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:30.735290 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" podStartSLOduration=1.735269693 podStartE2EDuration="1.735269693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:30.732964645 +0000 UTC m=+1979.414114533" watchObservedRunningTime="2026-10-30 22:15:30.735269693 +0000 UTC m=+1979.416419591" Oct 30 22:15:31.626231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:31.626195 4933 generic.go:356] "Generic (PLEG): container finished" podID="bf216553-f6e3-407d-9f44-2e2fd786f836" containerID="456f3c0597c47b86fd25ce5c16095b7a16f6d893211a4be409262c2ae92473cd" exitCode=0 Oct 30 22:15:31.626648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:31.626281 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" event={"ID":"bf216553-f6e3-407d-9f44-2e2fd786f836","Type":"ContainerDied","Data":"456f3c0597c47b86fd25ce5c16095b7a16f6d893211a4be409262c2ae92473cd"} Oct 30 22:15:32.811271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.811248 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:32.917401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.917337 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-openshift-ca\") pod \"bf216553-f6e3-407d-9f44-2e2fd786f836\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " Oct 30 22:15:32.917401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.917400 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf216553-f6e3-407d-9f44-2e2fd786f836-kube-api-access-5tqrh\" (UniqueName: \"kubernetes.io/projected/bf216553-f6e3-407d-9f44-2e2fd786f836-kube-api-access-5tqrh\") pod \"bf216553-f6e3-407d-9f44-2e2fd786f836\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " Oct 30 22:15:32.917653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.917456 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-xks-ca\") pod \"bf216553-f6e3-407d-9f44-2e2fd786f836\" (UID: \"bf216553-f6e3-407d-9f44-2e2fd786f836\") " Oct 30 22:15:32.917723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.917671 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-openshift-ca" pod "bf216553-f6e3-407d-9f44-2e2fd786f836" (UID: "bf216553-f6e3-407d-9f44-2e2fd786f836"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:32.917995 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.917974 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:32.919618 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.919592 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-xks-ca" pod "bf216553-f6e3-407d-9f44-2e2fd786f836" (UID: "bf216553-f6e3-407d-9f44-2e2fd786f836"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:32.919722 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:32.919656 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf216553-f6e3-407d-9f44-2e2fd786f836-kube-api-access-5tqrh" pod "bf216553-f6e3-407d-9f44-2e2fd786f836" (UID: "bf216553-f6e3-407d-9f44-2e2fd786f836"). InnerVolumeSpecName "kube-api-access-5tqrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:33.019813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.019787 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tqrh\" (UniqueName: \"kubernetes.io/projected/bf216553-f6e3-407d-9f44-2e2fd786f836-kube-api-access-5tqrh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:33.019813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.019813 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf216553-f6e3-407d-9f44-2e2fd786f836-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:33.578338 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.578303 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm"] Oct 30 22:15:33.628603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.628573 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk"] Oct 30 22:15:33.633302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.633262 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9"] Oct 30 22:15:33.648874 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.648848 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" Oct 30 22:15:33.649045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.648871 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf" event={"ID":"bf216553-f6e3-407d-9f44-2e2fd786f836","Type":"ContainerDied","Data":"c8209b051388ce3f0f46f62f70df13e149bb072d1b6b7679796cd0764b06fe9a"} Oct 30 22:15:33.649045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:33.648923 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8209b051388ce3f0f46f62f70df13e149bb072d1b6b7679796cd0764b06fe9a" Oct 30 22:15:40.719398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:40.719364 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn"] Oct 30 22:15:45.521101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521070 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t"] Oct 30 22:15:45.521702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521686 4933 cpu_manager.go:414] "containerMap: removing container" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:45.521756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521704 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:45.521756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521716 4933 cpu_manager.go:414] "containerMap: removing container" podUID="bf216553-f6e3-407d-9f44-2e2fd786f836" containerName="revoke-keys" Oct 30 22:15:45.521756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521721 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf216553-f6e3-407d-9f44-2e2fd786f836" containerName="revoke-keys" Oct 30 22:15:45.521756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521730 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:45.521756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521735 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:45.521756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521744 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:45.521756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521749 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521765 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521770 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1327a728-8c81-4c47-9985-397ff7342c4b" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521782 4933 cpu_manager.go:414] "containerMap: removing container" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521786 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521876 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c59b74-a98f-4464-b7e0-a228677cea1d" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521887 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c1f3f-7e42-488c-ba8c-36342be21470" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521892 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31c4d05-a234-4df5-a548-0574613c51f6" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521899 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0408ead9-3525-4ef7-a92f-17189843b02f" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521906 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d00942b-1e9e-4b7f-8ca7-91b88e153a8f" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521913 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d83313-5765-48db-9683-c852af36ad58" containerName="cleanup" Oct 30 22:15:45.521968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.521919 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf216553-f6e3-407d-9f44-2e2fd786f836" containerName="revoke-keys" Oct 30 22:15:45.526976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.526959 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.579870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.579849 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:45.585177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.585152 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t"] Oct 30 22:15:45.628652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.628629 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:45.628807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.628680 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:45.628807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.628743 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:15:45.721174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.721143 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" podUID="7f675717-5864-48b6-a2d3-88605c5558c7" containerName="payload-pre-processing" containerID="cri-o://9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66" gracePeriod=25 Oct 30 22:15:45.724208 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.724171 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.724366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.724275 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9m78\" (UniqueName: \"kubernetes.io/projected/883ce997-120f-4c76-97fa-b60eb5e81b71-kube-api-access-q9m78\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.724426 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.724409 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.776888 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.776858 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww"] Oct 30 22:15:45.777101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.777078 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" containerID="cri-o://523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614" gracePeriod=30 Oct 30 22:15:45.825748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.825712 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.825952 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.825793 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9m78\" (UniqueName: \"kubernetes.io/projected/883ce997-120f-4c76-97fa-b60eb5e81b71-kube-api-access-q9m78\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.825952 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.825906 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.826558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.826532 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.828686 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.828663 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:45.892303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:45.892245 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9m78\" (UniqueName: \"kubernetes.io/projected/883ce997-120f-4c76-97fa-b60eb5e81b71-kube-api-access-q9m78\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:46.140592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:46.140514 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:46.207537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:46.207461 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th"] Oct 30 22:15:46.218540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:46.208101 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" containerID="cri-o://0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00" gracePeriod=30 Oct 30 22:15:46.518628 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:46.518600 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883ce997_120f_4c76_97fa_b60eb5e81b71.slice/crio-533eb5877b9471f6fd0ec41836c410385ee28111be4ce68d1274fe26552ed467 WatchSource:0}: Error finding container 533eb5877b9471f6fd0ec41836c410385ee28111be4ce68d1274fe26552ed467: Status 404 returned error can't find the container with id 533eb5877b9471f6fd0ec41836c410385ee28111be4ce68d1274fe26552ed467 Oct 30 22:15:46.519937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:46.519914 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t"] Oct 30 22:15:46.543306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:46.543159 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:15:46.543306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:46.543219 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:15:46.795063 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:46.795026 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" event={"ID":"883ce997-120f-4c76-97fa-b60eb5e81b71","Type":"ContainerStarted","Data":"533eb5877b9471f6fd0ec41836c410385ee28111be4ce68d1274fe26552ed467"} Oct 30 22:15:47.622655 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:47.622620 4933 patch_prober.go:29] interesting pod/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:15:47.623007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:47.622680 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:15:47.804057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:47.804028 4933 generic.go:356] "Generic (PLEG): container finished" podID="883ce997-120f-4c76-97fa-b60eb5e81b71" containerID="2bf145195bed37e85f10986656d1333bb70752c72462d61538d3eafa06ddba45" exitCode=0 Oct 30 22:15:47.804231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:47.804111 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" event={"ID":"883ce997-120f-4c76-97fa-b60eb5e81b71","Type":"ContainerDied","Data":"2bf145195bed37e85f10986656d1333bb70752c72462d61538d3eafa06ddba45"} Oct 30 22:15:48.600282 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.600227 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" containerID="cri-o://825b22aad78ea35706bf0de3d4d1c7f213a0d19457da0951bde82b8ceb985569" gracePeriod=45 Oct 30 22:15:48.652450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.652410 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" containerID="cri-o://9277b773c79160900ff31206cdc2288f2f27102e64bf7331125082bd2ea46990" gracePeriod=45 Oct 30 22:15:48.655369 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.655334 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" containerID="cri-o://31c284a6a567103c5601d9d522c5d4f9040cb9fda8e4dbdbb33f1770df964d94" gracePeriod=45 Oct 30 22:15:48.735474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.735441 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused" Oct 30 22:15:48.817912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.817704 4933 generic.go:356] "Generic (PLEG): container finished" podID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerID="9277b773c79160900ff31206cdc2288f2f27102e64bf7331125082bd2ea46990" exitCode=0 Oct 30 22:15:48.817912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.817862 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" event={"ID":"f35b6179-b260-4be8-a5f2-0f94da37e2c9","Type":"ContainerDied","Data":"9277b773c79160900ff31206cdc2288f2f27102e64bf7331125082bd2ea46990"} Oct 30 22:15:48.820131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.819969 4933 generic.go:356] "Generic (PLEG): container finished" podID="c5363e34-af60-478e-ad07-07bb06305a80" containerID="825b22aad78ea35706bf0de3d4d1c7f213a0d19457da0951bde82b8ceb985569" exitCode=0 Oct 30 22:15:48.820131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.820075 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" event={"ID":"c5363e34-af60-478e-ad07-07bb06305a80","Type":"ContainerDied","Data":"825b22aad78ea35706bf0de3d4d1c7f213a0d19457da0951bde82b8ceb985569"} Oct 30 22:15:48.822007 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.821988 4933 generic.go:356] "Generic (PLEG): container finished" podID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerID="31c284a6a567103c5601d9d522c5d4f9040cb9fda8e4dbdbb33f1770df964d94" exitCode=0 Oct 30 22:15:48.822555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:48.822140 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" event={"ID":"9ef2246f-256c-4851-81e9-b8dbf577ff54","Type":"ContainerDied","Data":"31c284a6a567103c5601d9d522c5d4f9040cb9fda8e4dbdbb33f1770df964d94"} Oct 30 22:15:49.003437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.003081 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.095069 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-model-cache\") pod \"c5363e34-af60-478e-ad07-07bb06305a80\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.095156 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5363e34-af60-478e-ad07-07bb06305a80-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5363e34-af60-478e-ad07-07bb06305a80-tls-certs\") pod \"c5363e34-af60-478e-ad07-07bb06305a80\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.095194 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5363e34-af60-478e-ad07-07bb06305a80-kube-api-access-gvm7c\" (UniqueName: \"kubernetes.io/projected/c5363e34-af60-478e-ad07-07bb06305a80-kube-api-access-gvm7c\") pod \"c5363e34-af60-478e-ad07-07bb06305a80\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.095231 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-tmp-dir\") pod \"c5363e34-af60-478e-ad07-07bb06305a80\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.095260 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-dshm\") pod \"c5363e34-af60-478e-ad07-07bb06305a80\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.095318 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-home\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-home\") pod \"c5363e34-af60-478e-ad07-07bb06305a80\" (UID: \"c5363e34-af60-478e-ad07-07bb06305a80\") " Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.099758 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-home" pod "c5363e34-af60-478e-ad07-07bb06305a80" (UID: "c5363e34-af60-478e-ad07-07bb06305a80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.100373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.100036 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-model-cache" pod "c5363e34-af60-478e-ad07-07bb06305a80" (UID: "c5363e34-af60-478e-ad07-07bb06305a80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.104131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.103261 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-tmp-dir" pod "c5363e34-af60-478e-ad07-07bb06305a80" (UID: "c5363e34-af60-478e-ad07-07bb06305a80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.205637 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.200850 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.205637 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.200889 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.205637 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.200903 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.207515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.206991 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5363e34-af60-478e-ad07-07bb06305a80-kube-api-access-gvm7c" pod "c5363e34-af60-478e-ad07-07bb06305a80" (UID: "c5363e34-af60-478e-ad07-07bb06305a80"). InnerVolumeSpecName "kube-api-access-gvm7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:49.208663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.208631 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5363e34-af60-478e-ad07-07bb06305a80-tls-certs" pod "c5363e34-af60-478e-ad07-07bb06305a80" (UID: "c5363e34-af60-478e-ad07-07bb06305a80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:49.214404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.213944 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:15:49.223751 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.223667 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:15:49.246257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.235628 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-dshm" pod "c5363e34-af60-478e-ad07-07bb06305a80" (UID: "c5363e34-af60-478e-ad07-07bb06305a80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303026 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef2246f-256c-4851-81e9-b8dbf577ff54-kube-api-access-dfwdg\" (UniqueName: \"kubernetes.io/projected/9ef2246f-256c-4851-81e9-b8dbf577ff54-kube-api-access-dfwdg\") pod \"9ef2246f-256c-4851-81e9-b8dbf577ff54\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303067 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-home\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-home\") pod \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303100 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35b6179-b260-4be8-a5f2-0f94da37e2c9-kube-api-access-jxwjw\" (UniqueName: \"kubernetes.io/projected/f35b6179-b260-4be8-a5f2-0f94da37e2c9-kube-api-access-jxwjw\") pod \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303131 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-model-cache\") pod \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303158 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tmp-dir\") pod \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303188 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef2246f-256c-4851-81e9-b8dbf577ff54-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef2246f-256c-4851-81e9-b8dbf577ff54-tls-certs\") pod \"9ef2246f-256c-4851-81e9-b8dbf577ff54\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303214 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-dshm\") pod \"9ef2246f-256c-4851-81e9-b8dbf577ff54\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303279 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-dshm\") pod \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303309 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tls-certs\") pod \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\" (UID: \"f35b6179-b260-4be8-a5f2-0f94da37e2c9\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303331 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-home\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-home\") pod \"9ef2246f-256c-4851-81e9-b8dbf577ff54\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303362 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-tmp-dir\") pod \"9ef2246f-256c-4851-81e9-b8dbf577ff54\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.303409 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-model-cache\") pod \"9ef2246f-256c-4851-81e9-b8dbf577ff54\" (UID: \"9ef2246f-256c-4851-81e9-b8dbf577ff54\") " Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304121 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5363e34-af60-478e-ad07-07bb06305a80-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304147 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvm7c\" (UniqueName: \"kubernetes.io/projected/c5363e34-af60-478e-ad07-07bb06305a80-kube-api-access-gvm7c\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304171 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5363e34-af60-478e-ad07-07bb06305a80-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304235 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-model-cache" pod "f35b6179-b260-4be8-a5f2-0f94da37e2c9" (UID: "f35b6179-b260-4be8-a5f2-0f94da37e2c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304319 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-model-cache" pod "9ef2246f-256c-4851-81e9-b8dbf577ff54" (UID: "9ef2246f-256c-4851-81e9-b8dbf577ff54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304473 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tmp-dir" pod "f35b6179-b260-4be8-a5f2-0f94da37e2c9" (UID: "f35b6179-b260-4be8-a5f2-0f94da37e2c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304544 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-home" pod "9ef2246f-256c-4851-81e9-b8dbf577ff54" (UID: "9ef2246f-256c-4851-81e9-b8dbf577ff54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.304964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.304723 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-tmp-dir" pod "9ef2246f-256c-4851-81e9-b8dbf577ff54" (UID: "9ef2246f-256c-4851-81e9-b8dbf577ff54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.310015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.305258 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-home" pod "f35b6179-b260-4be8-a5f2-0f94da37e2c9" (UID: "f35b6179-b260-4be8-a5f2-0f94da37e2c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.310588 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.310559 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef2246f-256c-4851-81e9-b8dbf577ff54-kube-api-access-dfwdg" pod "9ef2246f-256c-4851-81e9-b8dbf577ff54" (UID: "9ef2246f-256c-4851-81e9-b8dbf577ff54"). InnerVolumeSpecName "kube-api-access-dfwdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:49.311403 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.311377 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-dshm" pod "f35b6179-b260-4be8-a5f2-0f94da37e2c9" (UID: "f35b6179-b260-4be8-a5f2-0f94da37e2c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.311908 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.311883 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35b6179-b260-4be8-a5f2-0f94da37e2c9-kube-api-access-jxwjw" pod "f35b6179-b260-4be8-a5f2-0f94da37e2c9" (UID: "f35b6179-b260-4be8-a5f2-0f94da37e2c9"). InnerVolumeSpecName "kube-api-access-jxwjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:49.312654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.312618 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-dshm" pod "9ef2246f-256c-4851-81e9-b8dbf577ff54" (UID: "9ef2246f-256c-4851-81e9-b8dbf577ff54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:49.313871 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.313845 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef2246f-256c-4851-81e9-b8dbf577ff54-tls-certs" pod "9ef2246f-256c-4851-81e9-b8dbf577ff54" (UID: "9ef2246f-256c-4851-81e9-b8dbf577ff54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:49.315317 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.315296 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tls-certs" pod "f35b6179-b260-4be8-a5f2-0f94da37e2c9" (UID: "f35b6179-b260-4be8-a5f2-0f94da37e2c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:49.370151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.370120 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:49.404888 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.404816 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883ce997-120f-4c76-97fa-b60eb5e81b71-kube-api-access-q9m78\" (UniqueName: \"kubernetes.io/projected/883ce997-120f-4c76-97fa-b60eb5e81b71-kube-api-access-q9m78\") pod \"883ce997-120f-4c76-97fa-b60eb5e81b71\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " Oct 30 22:15:49.405000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.404889 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-openshift-ca\") pod \"883ce997-120f-4c76-97fa-b60eb5e81b71\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " Oct 30 22:15:49.405105 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405092 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-xks-ca\") pod \"883ce997-120f-4c76-97fa-b60eb5e81b71\" (UID: \"883ce997-120f-4c76-97fa-b60eb5e81b71\") " Oct 30 22:15:49.405256 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405230 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-openshift-ca" pod "883ce997-120f-4c76-97fa-b60eb5e81b71" (UID: "883ce997-120f-4c76-97fa-b60eb5e81b71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:49.405504 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405471 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwdg\" (UniqueName: \"kubernetes.io/projected/9ef2246f-256c-4851-81e9-b8dbf577ff54-kube-api-access-dfwdg\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405509 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405518 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxwjw\" (UniqueName: \"kubernetes.io/projected/f35b6179-b260-4be8-a5f2-0f94da37e2c9-kube-api-access-jxwjw\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405527 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405540 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405553 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405568 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef2246f-256c-4851-81e9-b8dbf577ff54-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405580 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.405589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405589 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f35b6179-b260-4be8-a5f2-0f94da37e2c9-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.406189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405597 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f35b6179-b260-4be8-a5f2-0f94da37e2c9-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.406189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405604 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.406189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405612 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.406189 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.405619 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef2246f-256c-4851-81e9-b8dbf577ff54-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.407158 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.407132 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883ce997-120f-4c76-97fa-b60eb5e81b71-kube-api-access-q9m78" pod "883ce997-120f-4c76-97fa-b60eb5e81b71" (UID: "883ce997-120f-4c76-97fa-b60eb5e81b71"). InnerVolumeSpecName "kube-api-access-q9m78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:49.407338 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.407323 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-xks-ca" pod "883ce997-120f-4c76-97fa-b60eb5e81b71" (UID: "883ce997-120f-4c76-97fa-b60eb5e81b71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:49.507426 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.507399 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/883ce997-120f-4c76-97fa-b60eb5e81b71-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.507643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.507451 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9m78\" (UniqueName: \"kubernetes.io/projected/883ce997-120f-4c76-97fa-b60eb5e81b71-kube-api-access-q9m78\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:49.831124 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.831088 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" Oct 30 22:15:49.831662 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.831117 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm" event={"ID":"c5363e34-af60-478e-ad07-07bb06305a80","Type":"ContainerDied","Data":"16a714b778569022d38179403abbdde88ccb624dcc71203e5c1f226f95c6fedd"} Oct 30 22:15:49.831662 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.831177 4933 scope.go:122] "RemoveContainer" containerID="825b22aad78ea35706bf0de3d4d1c7f213a0d19457da0951bde82b8ceb985569" Oct 30 22:15:49.832939 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.832916 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" Oct 30 22:15:49.833067 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.832924 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9" event={"ID":"9ef2246f-256c-4851-81e9-b8dbf577ff54","Type":"ContainerDied","Data":"03d50a2169e340b2cb0d5c22faeab108759f24be36fe7d907d416a2465ac7a93"} Oct 30 22:15:49.834639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.834619 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" Oct 30 22:15:49.834750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.834657 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t" event={"ID":"883ce997-120f-4c76-97fa-b60eb5e81b71","Type":"ContainerDied","Data":"533eb5877b9471f6fd0ec41836c410385ee28111be4ce68d1274fe26552ed467"} Oct 30 22:15:49.834750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.834698 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="533eb5877b9471f6fd0ec41836c410385ee28111be4ce68d1274fe26552ed467" Oct 30 22:15:49.836602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.836582 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" event={"ID":"f35b6179-b260-4be8-a5f2-0f94da37e2c9","Type":"ContainerDied","Data":"2a948a258d204ed3d837bdb365446cb49493ed0f0341ec92615fb15040cb0cb1"} Oct 30 22:15:49.836602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.836589 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" Oct 30 22:15:49.850591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.849647 4933 scope.go:122] "RemoveContainer" containerID="31c284a6a567103c5601d9d522c5d4f9040cb9fda8e4dbdbb33f1770df964d94" Oct 30 22:15:49.862650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.862627 4933 scope.go:122] "RemoveContainer" containerID="9277b773c79160900ff31206cdc2288f2f27102e64bf7331125082bd2ea46990" Oct 30 22:15:49.942104 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:49.942060 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 30 22:15:50.052185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.052151 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk"] Oct 30 22:15:50.060061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.060037 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-w1-d516c4-kserve-9b8955b4c-xx2mk"] Oct 30 22:15:50.169580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.169467 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9"] Oct 30 22:15:50.173395 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.173369 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/simulator-w1-d516c4-kserve-8d55d8669-rzps9"] Oct 30 22:15:50.290449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.290411 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm"] Oct 30 22:15:50.295212 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.295179 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d516c4/distinct-2-w1-d516c4-kserve-95f94b69b-ld8tm"] Oct 30 22:15:50.403589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.403554 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4"] Oct 30 22:15:50.409575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:50.409552 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8f1d78db-9t5f4"] Oct 30 22:15:51.057710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.057655 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 22:15:51.129996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.129971 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-envoy\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130147 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130035 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-podinfo\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130147 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130089 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-certs\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130160 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-credential-socket\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130179 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7efe19-eacd-4488-81c6-44d87f02ac64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7efe19-eacd-4488-81c6-44d87f02ac64-istiod-ca-cert\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130218 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-data\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130250 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-kube-api-access-c9nk7\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-kube-api-access-c9nk7\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130371 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-socket\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130404 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-token\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-token\") pod \"0c7efe19-eacd-4488-81c6-44d87f02ac64\" (UID: \"0c7efe19-eacd-4488-81c6-44d87f02ac64\") " Oct 30 22:15:51.130648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130466 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-credential-socket" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.130648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130601 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7efe19-eacd-4488-81c6-44d87f02ac64-istiod-ca-cert" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:51.130978 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130758 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-data" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.130978 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.130907 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-socket" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.131093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.131026 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-certs" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.131177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.131155 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.131239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.131182 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.131239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.131196 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.131239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.131213 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c7efe19-eacd-4488-81c6-44d87f02ac64-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.131239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.131228 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.132525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.132480 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-podinfo" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:15:51.132652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.132567 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-envoy" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.132900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.132883 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-token" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:51.133072 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.133052 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-kube-api-access-c9nk7" pod "0c7efe19-eacd-4488-81c6-44d87f02ac64" (UID: "0c7efe19-eacd-4488-81c6-44d87f02ac64"). InnerVolumeSpecName "kube-api-access-c9nk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:51.233172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.233140 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.233302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.233176 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.233302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.233193 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c7efe19-eacd-4488-81c6-44d87f02ac64-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.233302 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.233208 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9nk7\" (UniqueName: \"kubernetes.io/projected/0c7efe19-eacd-4488-81c6-44d87f02ac64-kube-api-access-c9nk7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.478057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.478029 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3deada50-f885-44c5-a5b8-e46727f7b606" path="/var/lib/kubelet/pods/3deada50-f885-44c5-a5b8-e46727f7b606/volumes" Oct 30 22:15:51.478706 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.478680 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" path="/var/lib/kubelet/pods/9ef2246f-256c-4851-81e9-b8dbf577ff54/volumes" Oct 30 22:15:51.479172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.479150 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5363e34-af60-478e-ad07-07bb06305a80" path="/var/lib/kubelet/pods/c5363e34-af60-478e-ad07-07bb06305a80/volumes" Oct 30 22:15:51.480141 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.480120 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" path="/var/lib/kubelet/pods/f35b6179-b260-4be8-a5f2-0f94da37e2c9/volumes" Oct 30 22:15:51.489075 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.488901 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:15:51.538477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538445 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadd75e2-49fd-4adb-9128-d185937f4bc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadd75e2-49fd-4adb-9128-d185937f4bc1-istiod-ca-cert\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538513 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-socket\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538626 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-kube-api-access-pn5mk\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-kube-api-access-pn5mk\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538815 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538689 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-podinfo\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538815 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538768 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-token\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-token\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538815 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538790 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-certs\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538850 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-envoy\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538870 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-credential-socket\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.538976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538882 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-socket" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.538976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538891 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadd75e2-49fd-4adb-9128-d185937f4bc1-istiod-ca-cert" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:51.538976 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.538907 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-data\") pod \"cadd75e2-49fd-4adb-9128-d185937f4bc1\" (UID: \"cadd75e2-49fd-4adb-9128-d185937f4bc1\") " Oct 30 22:15:51.539237 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.539095 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-certs" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.539359 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.539336 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-data" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.539875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.539610 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cadd75e2-49fd-4adb-9128-d185937f4bc1-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.539875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.539638 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.539875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.539653 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.539875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.539743 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-credential-socket" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.539875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.539782 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.541442 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.541416 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-kube-api-access-pn5mk" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "kube-api-access-pn5mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:51.541716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.541696 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-podinfo" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:15:51.541794 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.541759 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-envoy" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:15:51.542214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.542193 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-token" pod "cadd75e2-49fd-4adb-9128-d185937f4bc1" (UID: "cadd75e2-49fd-4adb-9128-d185937f4bc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:51.641714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.641637 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn5mk\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-kube-api-access-pn5mk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.641714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.641673 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.641714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.641689 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.641714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.641701 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.641714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.641714 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cadd75e2-49fd-4adb-9128-d185937f4bc1-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:51.857961 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.857925 4933 generic.go:356] "Generic (PLEG): container finished" podID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerID="523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614" exitCode=0 Oct 30 22:15:51.858133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.857966 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" Oct 30 22:15:51.858133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.858006 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" event={"ID":"0c7efe19-eacd-4488-81c6-44d87f02ac64","Type":"ContainerDied","Data":"523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614"} Oct 30 22:15:51.858133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.858068 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww" event={"ID":"0c7efe19-eacd-4488-81c6-44d87f02ac64","Type":"ContainerDied","Data":"ea228fa2807422117174ee67403002596c483fb7ce4475599b87477f20bfbb2a"} Oct 30 22:15:51.858133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.858098 4933 scope.go:122] "RemoveContainer" containerID="523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614" Oct 30 22:15:51.859662 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.859629 4933 generic.go:356] "Generic (PLEG): container finished" podID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerID="0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00" exitCode=0 Oct 30 22:15:51.859784 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.859687 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" Oct 30 22:15:51.859784 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.859691 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" event={"ID":"cadd75e2-49fd-4adb-9128-d185937f4bc1","Type":"ContainerDied","Data":"0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00"} Oct 30 22:15:51.859784 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.859758 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th" event={"ID":"cadd75e2-49fd-4adb-9128-d185937f4bc1","Type":"ContainerDied","Data":"70b5bfe5c2fbde9403769c3bcaae420e0d1215efc0038f215972af0567eca5e9"} Oct 30 22:15:51.870261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.870245 4933 scope.go:122] "RemoveContainer" containerID="523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614" Oct 30 22:15:51.870587 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:51.870570 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614\": container with ID starting with 523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614 not found: ID does not exist" containerID="523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614" Oct 30 22:15:51.870660 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.870596 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614"} err="failed to get container status \"523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614\": rpc error: code = NotFound desc = could not find container \"523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614\": container with ID starting with 523fcf8b2eefeb5e97c9c5602cf59b4dc4544f063c3534202df4dd1ce56ff614 not found: ID does not exist" Oct 30 22:15:51.870660 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.870612 4933 scope.go:122] "RemoveContainer" containerID="0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00" Oct 30 22:15:51.880227 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.880208 4933 scope.go:122] "RemoveContainer" containerID="0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00" Oct 30 22:15:51.880500 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:51.880466 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00\": container with ID starting with 0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00 not found: ID does not exist" containerID="0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00" Oct 30 22:15:51.880561 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:51.880512 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00"} err="failed to get container status \"0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00\": rpc error: code = NotFound desc = could not find container \"0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00\": container with ID starting with 0b3efbccb345a85af70551a45cb08b901ed032be021923c007a41dae7ab47e00 not found: ID does not exist" Oct 30 22:15:52.026769 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:52.026741 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww"] Oct 30 22:15:52.077083 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:52.077054 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-rpsww"] Oct 30 22:15:52.096021 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:52.095994 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" podUID="7f675717-5864-48b6-a2d3-88605c5558c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:15:52.139594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:52.139571 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th"] Oct 30 22:15:52.147785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:52.147760 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-9q8th"] Oct 30 22:15:53.466649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:53.466609 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" path="/var/lib/kubelet/pods/0c7efe19-eacd-4488-81c6-44d87f02ac64/volumes" Oct 30 22:15:53.467249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:53.467226 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" path="/var/lib/kubelet/pods/cadd75e2-49fd-4adb-9128-d185937f4bc1/volumes" Oct 30 22:15:54.757546 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.757515 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x"] Oct 30 22:15:54.758089 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758075 4933 cpu_manager.go:414] "containerMap: removing container" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" Oct 30 22:15:54.758134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758089 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" Oct 30 22:15:54.758134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758104 4933 cpu_manager.go:414] "containerMap: removing container" podUID="883ce997-120f-4c76-97fa-b60eb5e81b71" containerName="revoke-keys" Oct 30 22:15:54.758134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758109 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883ce997-120f-4c76-97fa-b60eb5e81b71" containerName="revoke-keys" Oct 30 22:15:54.758134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758132 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758137 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758153 4933 cpu_manager.go:414] "containerMap: removing container" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758157 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758168 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758172 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758181 4933 cpu_manager.go:414] "containerMap: removing container" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758185 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" Oct 30 22:15:54.758265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758261 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef2246f-256c-4851-81e9-b8dbf577ff54" containerName="main" Oct 30 22:15:54.758545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758271 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5363e34-af60-478e-ad07-07bb06305a80" containerName="main" Oct 30 22:15:54.758545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758277 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7efe19-eacd-4488-81c6-44d87f02ac64" containerName="istio-proxy" Oct 30 22:15:54.758545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758284 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadd75e2-49fd-4adb-9128-d185937f4bc1" containerName="istio-proxy" Oct 30 22:15:54.758545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758290 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35b6179-b260-4be8-a5f2-0f94da37e2c9" containerName="main" Oct 30 22:15:54.758545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.758296 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="883ce997-120f-4c76-97fa-b60eb5e81b71" containerName="revoke-keys" Oct 30 22:15:54.762045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.762028 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.817475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.817440 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4"] Oct 30 22:15:54.822459 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.822438 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x"] Oct 30 22:15:54.822607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.822579 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.864567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.864240 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:15:54.864567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.864322 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:54.868419 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.868397 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:54.868561 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.868460 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:15:54.880331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.880304 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4"] Oct 30 22:15:54.883768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.883744 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.883890 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.883862 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.883982 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.883901 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6b89\" (UniqueName: \"kubernetes.io/projected/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-kube-api-access-j6b89\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.884046 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.884004 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.884102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.884064 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86jhp\" (UniqueName: \"kubernetes.io/projected/d65fdb8d-a940-4e8c-b589-f4254f5104da-kube-api-access-86jhp\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.884190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.884176 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.985437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.985409 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.985437 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.985440 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6b89\" (UniqueName: \"kubernetes.io/projected/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-kube-api-access-j6b89\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.985742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.985519 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.985742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.985568 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86jhp\" (UniqueName: \"kubernetes.io/projected/d65fdb8d-a940-4e8c-b589-f4254f5104da-kube-api-access-86jhp\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.985742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.985673 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.985742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.985719 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.986174 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.986155 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.986326 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.986305 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:54.988166 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.988145 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:54.988255 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:54.988203 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:55.043416 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.043386 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6b89\" (UniqueName: \"kubernetes.io/projected/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-kube-api-access-j6b89\") pod \"maas-api-revoke-sub-0ebbfe6e-fwt5x\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:55.043597 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.043524 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86jhp\" (UniqueName: \"kubernetes.io/projected/d65fdb8d-a940-4e8c-b589-f4254f5104da-kube-api-access-86jhp\") pod \"maas-api-revoke-sub-7b161ac3-4zjv4\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:55.075561 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.075529 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:55.137972 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.137940 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:55.453998 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:55.453965 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5137afc6_0d4c_4cc3_80c0_af971c57aa4c.slice/crio-723b82951572455283af44a0c3e89614dce93ce48932321337768836acff6cda WatchSource:0}: Error finding container 723b82951572455283af44a0c3e89614dce93ce48932321337768836acff6cda: Status 404 returned error can't find the container with id 723b82951572455283af44a0c3e89614dce93ce48932321337768836acff6cda Oct 30 22:15:55.454328 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.454306 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x"] Oct 30 22:15:55.563358 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:55.563334 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65fdb8d_a940_4e8c_b589_f4254f5104da.slice/crio-cf5413d9048b12e2726a6fdc50b9ec6de5337002dc41cce6e5f52174b99f8f0f WatchSource:0}: Error finding container cf5413d9048b12e2726a6fdc50b9ec6de5337002dc41cce6e5f52174b99f8f0f: Status 404 returned error can't find the container with id cf5413d9048b12e2726a6fdc50b9ec6de5337002dc41cce6e5f52174b99f8f0f Oct 30 22:15:55.564738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.564714 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4"] Oct 30 22:15:55.901959 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.901924 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" event={"ID":"5137afc6-0d4c-4cc3-80c0-af971c57aa4c","Type":"ContainerStarted","Data":"1cfb1d067705327c3719cc8672732a80bb2923077bf33b986842519617640bca"} Oct 30 22:15:55.902368 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.901985 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" event={"ID":"5137afc6-0d4c-4cc3-80c0-af971c57aa4c","Type":"ContainerStarted","Data":"723b82951572455283af44a0c3e89614dce93ce48932321337768836acff6cda"} Oct 30 22:15:55.903353 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:55.903328 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" event={"ID":"d65fdb8d-a940-4e8c-b589-f4254f5104da","Type":"ContainerStarted","Data":"cf5413d9048b12e2726a6fdc50b9ec6de5337002dc41cce6e5f52174b99f8f0f"} Oct 30 22:15:56.020842 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:56.020790 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" podStartSLOduration=2.020775607 podStartE2EDuration="2.020775607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:56.017399577 +0000 UTC m=+2004.698549472" watchObservedRunningTime="2026-10-30 22:15:56.020775607 +0000 UTC m=+2004.701925499" Oct 30 22:15:56.912445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:56.912356 4933 generic.go:356] "Generic (PLEG): container finished" podID="5137afc6-0d4c-4cc3-80c0-af971c57aa4c" containerID="1cfb1d067705327c3719cc8672732a80bb2923077bf33b986842519617640bca" exitCode=0 Oct 30 22:15:56.912879 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:56.912435 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" event={"ID":"5137afc6-0d4c-4cc3-80c0-af971c57aa4c","Type":"ContainerDied","Data":"1cfb1d067705327c3719cc8672732a80bb2923077bf33b986842519617640bca"} Oct 30 22:15:56.914405 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:56.914377 4933 generic.go:356] "Generic (PLEG): container finished" podID="d65fdb8d-a940-4e8c-b589-f4254f5104da" containerID="629dfa400845e9e75e1143271a5d50d90d2f4b61d81c7875c7c6a0e546b28bbd" exitCode=0 Oct 30 22:15:56.914568 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:56.914452 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" event={"ID":"d65fdb8d-a940-4e8c-b589-f4254f5104da","Type":"ContainerDied","Data":"629dfa400845e9e75e1143271a5d50d90d2f4b61d81c7875c7c6a0e546b28bbd"} Oct 30 22:15:58.255478 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.255455 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:58.259751 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.259724 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:58.364470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.364425 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65fdb8d-a940-4e8c-b589-f4254f5104da-kube-api-access-86jhp\" (UniqueName: \"kubernetes.io/projected/d65fdb8d-a940-4e8c-b589-f4254f5104da-kube-api-access-86jhp\") pod \"d65fdb8d-a940-4e8c-b589-f4254f5104da\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " Oct 30 22:15:58.364703 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.364616 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-openshift-ca\") pod \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " Oct 30 22:15:58.364703 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.364660 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-openshift-ca\") pod \"d65fdb8d-a940-4e8c-b589-f4254f5104da\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " Oct 30 22:15:58.364821 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.364717 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-kube-api-access-j6b89\" (UniqueName: \"kubernetes.io/projected/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-kube-api-access-j6b89\") pod \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " Oct 30 22:15:58.364821 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.364775 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-xks-ca\") pod \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\" (UID: \"5137afc6-0d4c-4cc3-80c0-af971c57aa4c\") " Oct 30 22:15:58.364928 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.364833 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-xks-ca\") pod \"d65fdb8d-a940-4e8c-b589-f4254f5104da\" (UID: \"d65fdb8d-a940-4e8c-b589-f4254f5104da\") " Oct 30 22:15:58.364988 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.364964 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-openshift-ca" pod "5137afc6-0d4c-4cc3-80c0-af971c57aa4c" (UID: "5137afc6-0d4c-4cc3-80c0-af971c57aa4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:58.365062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.365042 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-openshift-ca" pod "d65fdb8d-a940-4e8c-b589-f4254f5104da" (UID: "d65fdb8d-a940-4e8c-b589-f4254f5104da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:58.365462 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.365428 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:58.365462 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.365455 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:58.367603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.367576 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-xks-ca" pod "d65fdb8d-a940-4e8c-b589-f4254f5104da" (UID: "d65fdb8d-a940-4e8c-b589-f4254f5104da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:58.367730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.367695 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-kube-api-access-j6b89" pod "5137afc6-0d4c-4cc3-80c0-af971c57aa4c" (UID: "5137afc6-0d4c-4cc3-80c0-af971c57aa4c"). InnerVolumeSpecName "kube-api-access-j6b89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:58.367878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.367855 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-xks-ca" pod "5137afc6-0d4c-4cc3-80c0-af971c57aa4c" (UID: "5137afc6-0d4c-4cc3-80c0-af971c57aa4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:15:58.371906 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.371875 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65fdb8d-a940-4e8c-b589-f4254f5104da-kube-api-access-86jhp" pod "d65fdb8d-a940-4e8c-b589-f4254f5104da" (UID: "d65fdb8d-a940-4e8c-b589-f4254f5104da"). InnerVolumeSpecName "kube-api-access-86jhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:58.466891 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.466859 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65fdb8d-a940-4e8c-b589-f4254f5104da-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:58.467015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.466894 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86jhp\" (UniqueName: \"kubernetes.io/projected/d65fdb8d-a940-4e8c-b589-f4254f5104da-kube-api-access-86jhp\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:58.467015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.466913 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6b89\" (UniqueName: \"kubernetes.io/projected/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-kube-api-access-j6b89\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:58.467015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.466928 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5137afc6-0d4c-4cc3-80c0-af971c57aa4c-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:15:58.551049 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:15:58.551009 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:15:58.794542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.794512 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s"] Oct 30 22:15:58.810465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.810369 4933 cpu_manager.go:414] "containerMap: removing container" podUID="5137afc6-0d4c-4cc3-80c0-af971c57aa4c" containerName="revoke-keys" Oct 30 22:15:58.810750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.810725 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5137afc6-0d4c-4cc3-80c0-af971c57aa4c" containerName="revoke-keys" Oct 30 22:15:58.810994 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.810976 4933 cpu_manager.go:414] "containerMap: removing container" podUID="d65fdb8d-a940-4e8c-b589-f4254f5104da" containerName="revoke-keys" Oct 30 22:15:58.811158 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.811145 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65fdb8d-a940-4e8c-b589-f4254f5104da" containerName="revoke-keys" Oct 30 22:15:58.811536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.811518 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="5137afc6-0d4c-4cc3-80c0-af971c57aa4c" containerName="revoke-keys" Oct 30 22:15:58.811687 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.811673 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65fdb8d-a940-4e8c-b589-f4254f5104da" containerName="revoke-keys" Oct 30 22:15:58.826449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.826427 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:58.847424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.847392 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s"] Oct 30 22:15:58.978882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.978847 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:58.979193 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.979176 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:58.979329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.979315 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gkh\" (UniqueName: \"kubernetes.io/projected/9605d674-61da-49b6-9c86-bfeaf4e3a03f-kube-api-access-g4gkh\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:58.989810 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.989779 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" event={"ID":"5137afc6-0d4c-4cc3-80c0-af971c57aa4c","Type":"ContainerDied","Data":"723b82951572455283af44a0c3e89614dce93ce48932321337768836acff6cda"} Oct 30 22:15:58.989955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.989836 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="723b82951572455283af44a0c3e89614dce93ce48932321337768836acff6cda" Oct 30 22:15:58.989955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.989910 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x" Oct 30 22:15:58.991900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.991873 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" event={"ID":"d65fdb8d-a940-4e8c-b589-f4254f5104da","Type":"ContainerDied","Data":"cf5413d9048b12e2726a6fdc50b9ec6de5337002dc41cce6e5f52174b99f8f0f"} Oct 30 22:15:58.992026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.991917 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf5413d9048b12e2726a6fdc50b9ec6de5337002dc41cce6e5f52174b99f8f0f" Oct 30 22:15:58.992026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:58.991877 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4" Oct 30 22:15:59.087571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.083366 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:59.087571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.083450 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:59.087571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.083519 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gkh\" (UniqueName: \"kubernetes.io/projected/9605d674-61da-49b6-9c86-bfeaf4e3a03f-kube-api-access-g4gkh\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:59.087571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.084787 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:59.101165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.098059 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:59.150765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.147147 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gkh\" (UniqueName: \"kubernetes.io/projected/9605d674-61da-49b6-9c86-bfeaf4e3a03f-kube-api-access-g4gkh\") pod \"maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:59.150765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.149638 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:15:59.535353 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:15:59.535325 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9605d674_61da_49b6_9c86_bfeaf4e3a03f.slice/crio-5eb85aac2c0fc61a24f843070449119c84d994ff2c7dafb7ce99c98ce99c0c28 WatchSource:0}: Error finding container 5eb85aac2c0fc61a24f843070449119c84d994ff2c7dafb7ce99c98ce99c0c28: Status 404 returned error can't find the container with id 5eb85aac2c0fc61a24f843070449119c84d994ff2c7dafb7ce99c98ce99c0c28 Oct 30 22:15:59.535713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:15:59.535678 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s"] Oct 30 22:16:00.001107 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.001070 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" event={"ID":"9605d674-61da-49b6-9c86-bfeaf4e3a03f","Type":"ContainerStarted","Data":"4f5b73ea8149d09510609921b79e800abe41663648ef0eab307e3505a5561a23"} Oct 30 22:16:00.001276 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.001136 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" event={"ID":"9605d674-61da-49b6-9c86-bfeaf4e3a03f","Type":"ContainerStarted","Data":"5eb85aac2c0fc61a24f843070449119c84d994ff2c7dafb7ce99c98ce99c0c28"} Oct 30 22:16:00.119101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.119011 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" podStartSLOduration=2.118997841 podStartE2EDuration="2.118997841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:00.116616853 +0000 UTC m=+2008.797766747" watchObservedRunningTime="2026-10-30 22:16:00.118997841 +0000 UTC m=+2008.800147733" Oct 30 22:16:00.342292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.342261 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq"] Oct 30 22:16:00.355026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.354993 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.406721 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.406656 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq"] Oct 30 22:16:00.499108 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.499079 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g699d\" (UniqueName: \"kubernetes.io/projected/89e01147-e64d-4c5e-b2e6-27d18a84e378-kube-api-access-g699d\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.499231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.499210 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.499376 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.499279 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.600098 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.600071 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g699d\" (UniqueName: \"kubernetes.io/projected/89e01147-e64d-4c5e-b2e6-27d18a84e378-kube-api-access-g699d\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.600577 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.600159 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.600577 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.600230 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.600912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.600894 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.602649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.602631 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.658042 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.657963 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g699d\" (UniqueName: \"kubernetes.io/projected/89e01147-e64d-4c5e-b2e6-27d18a84e378-kube-api-access-g699d\") pod \"maas-api-revoke-sub-9b9db884-c6fmq\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:00.672727 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:00.672700 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:01.010623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:01.010538 4933 generic.go:356] "Generic (PLEG): container finished" podID="9605d674-61da-49b6-9c86-bfeaf4e3a03f" containerID="4f5b73ea8149d09510609921b79e800abe41663648ef0eab307e3505a5561a23" exitCode=0 Oct 30 22:16:01.010768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:01.010621 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" event={"ID":"9605d674-61da-49b6-9c86-bfeaf4e3a03f","Type":"ContainerDied","Data":"4f5b73ea8149d09510609921b79e800abe41663648ef0eab307e3505a5561a23"} Oct 30 22:16:01.076365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:01.076324 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq"] Oct 30 22:16:01.077198 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:01.077173 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e01147_e64d_4c5e_b2e6_27d18a84e378.slice/crio-a4778037723052ef1c977f5c2c68af5f46bb8f0e22d47eb14284183721d143c8 WatchSource:0}: Error finding container a4778037723052ef1c977f5c2c68af5f46bb8f0e22d47eb14284183721d143c8: Status 404 returned error can't find the container with id a4778037723052ef1c977f5c2c68af5f46bb8f0e22d47eb14284183721d143c8 Oct 30 22:16:02.020607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.020572 4933 generic.go:356] "Generic (PLEG): container finished" podID="89e01147-e64d-4c5e-b2e6-27d18a84e378" containerID="23155714b710e75be9cd3de626b0c2dbbedbc2942ca40d0b003971982af602c1" exitCode=0 Oct 30 22:16:02.021032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.020656 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" event={"ID":"89e01147-e64d-4c5e-b2e6-27d18a84e378","Type":"ContainerDied","Data":"23155714b710e75be9cd3de626b0c2dbbedbc2942ca40d0b003971982af602c1"} Oct 30 22:16:02.021032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.020708 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" event={"ID":"89e01147-e64d-4c5e-b2e6-27d18a84e378","Type":"ContainerStarted","Data":"a4778037723052ef1c977f5c2c68af5f46bb8f0e22d47eb14284183721d143c8"} Oct 30 22:16:02.098043 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.097981 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" podUID="7f675717-5864-48b6-a2d3-88605c5558c7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:16:02.203146 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.203124 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:16:02.322645 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.322617 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-xks-ca\") pod \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " Oct 30 22:16:02.322825 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.322689 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-openshift-ca\") pod \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " Oct 30 22:16:02.322825 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.322803 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9605d674-61da-49b6-9c86-bfeaf4e3a03f-kube-api-access-g4gkh\" (UniqueName: \"kubernetes.io/projected/9605d674-61da-49b6-9c86-bfeaf4e3a03f-kube-api-access-g4gkh\") pod \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\" (UID: \"9605d674-61da-49b6-9c86-bfeaf4e3a03f\") " Oct 30 22:16:02.323125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.323102 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-openshift-ca" pod "9605d674-61da-49b6-9c86-bfeaf4e3a03f" (UID: "9605d674-61da-49b6-9c86-bfeaf4e3a03f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:02.323434 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.323415 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:02.325025 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.325003 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-xks-ca" pod "9605d674-61da-49b6-9c86-bfeaf4e3a03f" (UID: "9605d674-61da-49b6-9c86-bfeaf4e3a03f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:02.325466 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.325433 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9605d674-61da-49b6-9c86-bfeaf4e3a03f-kube-api-access-g4gkh" pod "9605d674-61da-49b6-9c86-bfeaf4e3a03f" (UID: "9605d674-61da-49b6-9c86-bfeaf4e3a03f"). InnerVolumeSpecName "kube-api-access-g4gkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:02.425000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.424923 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9605d674-61da-49b6-9c86-bfeaf4e3a03f-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:02.425000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:02.424957 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4gkh\" (UniqueName: \"kubernetes.io/projected/9605d674-61da-49b6-9c86-bfeaf4e3a03f-kube-api-access-g4gkh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:03.028756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.028727 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" Oct 30 22:16:03.029141 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.028754 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s" event={"ID":"9605d674-61da-49b6-9c86-bfeaf4e3a03f","Type":"ContainerDied","Data":"5eb85aac2c0fc61a24f843070449119c84d994ff2c7dafb7ce99c98ce99c0c28"} Oct 30 22:16:03.029141 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.028797 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eb85aac2c0fc61a24f843070449119c84d994ff2c7dafb7ce99c98ce99c0c28" Oct 30 22:16:03.205612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.205589 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:03.223111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.223088 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t"] Oct 30 22:16:03.234886 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.234862 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf"] Oct 30 22:16:03.337029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.336950 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e01147-e64d-4c5e-b2e6-27d18a84e378-kube-api-access-g699d\" (UniqueName: \"kubernetes.io/projected/89e01147-e64d-4c5e-b2e6-27d18a84e378-kube-api-access-g699d\") pod \"89e01147-e64d-4c5e-b2e6-27d18a84e378\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " Oct 30 22:16:03.337029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.337014 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-xks-ca\") pod \"89e01147-e64d-4c5e-b2e6-27d18a84e378\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " Oct 30 22:16:03.337274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.337058 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-openshift-ca\") pod \"89e01147-e64d-4c5e-b2e6-27d18a84e378\" (UID: \"89e01147-e64d-4c5e-b2e6-27d18a84e378\") " Oct 30 22:16:03.338070 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.338043 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-openshift-ca" pod "89e01147-e64d-4c5e-b2e6-27d18a84e378" (UID: "89e01147-e64d-4c5e-b2e6-27d18a84e378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:03.340224 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.340195 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-xks-ca" pod "89e01147-e64d-4c5e-b2e6-27d18a84e378" (UID: "89e01147-e64d-4c5e-b2e6-27d18a84e378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:03.340525 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.340462 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e01147-e64d-4c5e-b2e6-27d18a84e378-kube-api-access-g699d" pod "89e01147-e64d-4c5e-b2e6-27d18a84e378" (UID: "89e01147-e64d-4c5e-b2e6-27d18a84e378"). InnerVolumeSpecName "kube-api-access-g699d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:03.438794 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.438766 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g699d\" (UniqueName: \"kubernetes.io/projected/89e01147-e64d-4c5e-b2e6-27d18a84e378-kube-api-access-g699d\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:03.438794 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.438792 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:03.438977 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:03.438802 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e01147-e64d-4c5e-b2e6-27d18a84e378-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:04.040802 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:04.040777 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" Oct 30 22:16:04.041198 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:04.040820 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq" event={"ID":"89e01147-e64d-4c5e-b2e6-27d18a84e378","Type":"ContainerDied","Data":"a4778037723052ef1c977f5c2c68af5f46bb8f0e22d47eb14284183721d143c8"} Oct 30 22:16:04.041198 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:04.040908 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4778037723052ef1c977f5c2c68af5f46bb8f0e22d47eb14284183721d143c8" Oct 30 22:16:06.028931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.028904 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 22:16:06.061932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.061898 4933 generic.go:356] "Generic (PLEG): container finished" podID="7f675717-5864-48b6-a2d3-88605c5558c7" containerID="9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66" exitCode=0 Oct 30 22:16:06.062096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.061934 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" Oct 30 22:16:06.062096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.061979 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" event={"ID":"7f675717-5864-48b6-a2d3-88605c5558c7","Type":"ContainerDied","Data":"9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66"} Oct 30 22:16:06.062096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.062034 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn" event={"ID":"7f675717-5864-48b6-a2d3-88605c5558c7","Type":"ContainerDied","Data":"f6e35b2a768ca76aca3572427e079651070a01b387cfd80fbca854fa3c9901a6"} Oct 30 22:16:06.062096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.062049 4933 scope.go:122] "RemoveContainer" containerID="9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66" Oct 30 22:16:06.074338 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.074307 4933 scope.go:122] "RemoveContainer" containerID="9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66" Oct 30 22:16:06.074704 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:06.074681 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66\": container with ID starting with 9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66 not found: ID does not exist" containerID="9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66" Oct 30 22:16:06.074871 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.074844 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66"} err="failed to get container status \"9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66\": rpc error: code = NotFound desc = could not find container \"9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66\": container with ID starting with 9f9b7d8d2efcb85985b830845746f742ba38759ecf915685f8385bdfe96e8e66 not found: ID does not exist" Oct 30 22:16:06.172916 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.172879 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f675717-5864-48b6-a2d3-88605c5558c7-kube-api-access-zrtdv\" (UniqueName: \"kubernetes.io/projected/7f675717-5864-48b6-a2d3-88605c5558c7-kube-api-access-zrtdv\") pod \"7f675717-5864-48b6-a2d3-88605c5558c7\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " Oct 30 22:16:06.173107 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.173050 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f675717-5864-48b6-a2d3-88605c5558c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f675717-5864-48b6-a2d3-88605c5558c7-plugins-config-volume\") pod \"7f675717-5864-48b6-a2d3-88605c5558c7\" (UID: \"7f675717-5864-48b6-a2d3-88605c5558c7\") " Oct 30 22:16:06.173401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.173373 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f675717-5864-48b6-a2d3-88605c5558c7-plugins-config-volume" pod "7f675717-5864-48b6-a2d3-88605c5558c7" (UID: "7f675717-5864-48b6-a2d3-88605c5558c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:06.173772 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.173750 4933 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f675717-5864-48b6-a2d3-88605c5558c7-plugins-config-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.175837 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.175804 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f675717-5864-48b6-a2d3-88605c5558c7-kube-api-access-zrtdv" pod "7f675717-5864-48b6-a2d3-88605c5558c7" (UID: "7f675717-5864-48b6-a2d3-88605c5558c7"). InnerVolumeSpecName "kube-api-access-zrtdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:06.275550 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.275521 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtdv\" (UniqueName: \"kubernetes.io/projected/7f675717-5864-48b6-a2d3-88605c5558c7-kube-api-access-zrtdv\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:06.509200 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.509158 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn"] Oct 30 22:16:06.515723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.515693 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d516c4-858d5d5b47-j2xfn"] Oct 30 22:16:06.704343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:06.704250 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:16:07.466832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:07.466802 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f675717-5864-48b6-a2d3-88605c5558c7" path="/var/lib/kubelet/pods/7f675717-5864-48b6-a2d3-88605c5558c7/volumes" Oct 30 22:16:15.665170 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665092 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts"] Oct 30 22:16:15.665702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665687 4933 cpu_manager.go:414] "containerMap: removing container" podUID="9605d674-61da-49b6-9c86-bfeaf4e3a03f" containerName="revoke-keys" Oct 30 22:16:15.665748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665703 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9605d674-61da-49b6-9c86-bfeaf4e3a03f" containerName="revoke-keys" Oct 30 22:16:15.665748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665716 4933 cpu_manager.go:414] "containerMap: removing container" podUID="89e01147-e64d-4c5e-b2e6-27d18a84e378" containerName="revoke-keys" Oct 30 22:16:15.665748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665721 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e01147-e64d-4c5e-b2e6-27d18a84e378" containerName="revoke-keys" Oct 30 22:16:15.665748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665728 4933 cpu_manager.go:414] "containerMap: removing container" podUID="7f675717-5864-48b6-a2d3-88605c5558c7" containerName="payload-pre-processing" Oct 30 22:16:15.665748 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665734 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f675717-5864-48b6-a2d3-88605c5558c7" containerName="payload-pre-processing" Oct 30 22:16:15.665905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665841 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="9605d674-61da-49b6-9c86-bfeaf4e3a03f" containerName="revoke-keys" Oct 30 22:16:15.665905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665849 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e01147-e64d-4c5e-b2e6-27d18a84e378" containerName="revoke-keys" Oct 30 22:16:15.665905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.665854 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f675717-5864-48b6-a2d3-88605c5558c7" containerName="payload-pre-processing" Oct 30 22:16:15.670653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.670639 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.699959 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.699936 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls"] Oct 30 22:16:15.703983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.703968 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.723809 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.723784 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:15.723963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.723945 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:16:15.754311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.750357 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:15.754311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.751290 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:15.769528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.769472 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.769670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.769585 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.769670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.769650 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85jz\" (UniqueName: \"kubernetes.io/projected/b67346a3-27eb-41a9-871a-9947ad851b1a-kube-api-access-q85jz\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.769894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.769685 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfljj\" (UniqueName: \"kubernetes.io/projected/cfcd3373-7782-4c15-a862-12c28d631416-kube-api-access-zfljj\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.769894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.769793 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.769894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.769867 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.782996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.782970 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts"] Oct 30 22:16:15.789590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.789564 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls"] Oct 30 22:16:15.871544 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.871470 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.871741 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.871652 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.871741 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.871710 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.871876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.871763 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85jz\" (UniqueName: \"kubernetes.io/projected/b67346a3-27eb-41a9-871a-9947ad851b1a-kube-api-access-q85jz\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.871876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.871795 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfljj\" (UniqueName: \"kubernetes.io/projected/cfcd3373-7782-4c15-a862-12c28d631416-kube-api-access-zfljj\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.871992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.871890 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.872716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.872690 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.872863 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.872736 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.875110 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.875084 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.875253 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.875230 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.936949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.936880 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfljj\" (UniqueName: \"kubernetes.io/projected/cfcd3373-7782-4c15-a862-12c28d631416-kube-api-access-zfljj\") pod \"maas-api-revoke-sub-fd8213aa-ctvts\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:15.937450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.937427 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85jz\" (UniqueName: \"kubernetes.io/projected/b67346a3-27eb-41a9-871a-9947ad851b1a-kube-api-access-q85jz\") pod \"maas-api-revoke-sub-28e69f00-wr9ls\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:15.986307 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:15.984739 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:16.020481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:16.019741 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:16.620984 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:16.620953 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcd3373_7782_4c15_a862_12c28d631416.slice/crio-2564df3d27960af29a2e34417477c0eb0d67a67bd851596a6b03f5b29d61921a WatchSource:0}: Error finding container 2564df3d27960af29a2e34417477c0eb0d67a67bd851596a6b03f5b29d61921a: Status 404 returned error can't find the container with id 2564df3d27960af29a2e34417477c0eb0d67a67bd851596a6b03f5b29d61921a Oct 30 22:16:16.621298 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:16.621281 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts"] Oct 30 22:16:16.671055 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:16.671027 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67346a3_27eb_41a9_871a_9947ad851b1a.slice/crio-7530ed8fef34ebe65624f446b07ac6d5a4bf3670fe6d3fc9f989eadf286bb95e WatchSource:0}: Error finding container 7530ed8fef34ebe65624f446b07ac6d5a4bf3670fe6d3fc9f989eadf286bb95e: Status 404 returned error can't find the container with id 7530ed8fef34ebe65624f446b07ac6d5a4bf3670fe6d3fc9f989eadf286bb95e Oct 30 22:16:16.673027 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:16.672963 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls"] Oct 30 22:16:17.342766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.342729 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" event={"ID":"cfcd3373-7782-4c15-a862-12c28d631416","Type":"ContainerStarted","Data":"e7ae9d7656f5ce123d3457831b8453e82ddf5cadd4588e605bfba8e73c5b9c81"} Oct 30 22:16:17.342952 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.342791 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" event={"ID":"cfcd3373-7782-4c15-a862-12c28d631416","Type":"ContainerStarted","Data":"2564df3d27960af29a2e34417477c0eb0d67a67bd851596a6b03f5b29d61921a"} Oct 30 22:16:17.344801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.344773 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" event={"ID":"b67346a3-27eb-41a9-871a-9947ad851b1a","Type":"ContainerStarted","Data":"7fa71253ad89140d9bffe95a48aaf5aaa465c1bc4d977391c239554786da1ab4"} Oct 30 22:16:17.344946 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.344818 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" event={"ID":"b67346a3-27eb-41a9-871a-9947ad851b1a","Type":"ContainerStarted","Data":"7530ed8fef34ebe65624f446b07ac6d5a4bf3670fe6d3fc9f989eadf286bb95e"} Oct 30 22:16:17.460908 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.460853 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" podStartSLOduration=2.460840103 podStartE2EDuration="2.460840103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:17.459913542 +0000 UTC m=+2026.141063441" watchObservedRunningTime="2026-10-30 22:16:17.460840103 +0000 UTC m=+2026.141989995" Oct 30 22:16:17.579695 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.579642 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" podStartSLOduration=2.5796276369999998 podStartE2EDuration="2.579627637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:17.575954381 +0000 UTC m=+2026.257104275" watchObservedRunningTime="2026-10-30 22:16:17.579627637 +0000 UTC m=+2026.260777529" Oct 30 22:16:17.939160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.939070 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg"] Oct 30 22:16:17.943373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.943350 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:17.982165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.982140 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm"] Oct 30 22:16:17.990327 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.990306 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7c826c-b5bjm"] Oct 30 22:16:17.997053 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:17.997032 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-5dff71-openshift-default-dockercfg-lfwp9\"" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.033660 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg"] Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035180 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035220 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035251 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035340 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7jq\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-kube-api-access-zk7jq\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035433 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035518 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035591 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035683 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.036527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.035740 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.136574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136540 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.136775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136579 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.136775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136677 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.136775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136714 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.136775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136759 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136871 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136899 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136932 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.136966 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7jq\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-kube-api-access-zk7jq\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.137098 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-data\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.137159 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-credential-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137566 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.137526 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-socket\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137566 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.137559 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istiod-ca-cert\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.137740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.137578 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-certs\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.139708 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.139680 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-envoy\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.139811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.139761 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-podinfo\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.201221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.199176 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-token\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.201221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.200728 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7jq\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-kube-api-access-zk7jq\") pod \"e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.250541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.245862 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" containerID="cri-o://379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478" gracePeriod=45 Oct 30 22:16:18.259544 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.259295 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:18.261502 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.261177 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" containerID="cri-o://9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f" gracePeriod=45 Oct 30 22:16:18.376213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.375694 4933 generic.go:356] "Generic (PLEG): container finished" podID="cfcd3373-7782-4c15-a862-12c28d631416" containerID="e7ae9d7656f5ce123d3457831b8453e82ddf5cadd4588e605bfba8e73c5b9c81" exitCode=0 Oct 30 22:16:18.376213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.375846 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" event={"ID":"cfcd3373-7782-4c15-a862-12c28d631416","Type":"ContainerDied","Data":"e7ae9d7656f5ce123d3457831b8453e82ddf5cadd4588e605bfba8e73c5b9c81"} Oct 30 22:16:18.379814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.379356 4933 generic.go:356] "Generic (PLEG): container finished" podID="b67346a3-27eb-41a9-871a-9947ad851b1a" containerID="7fa71253ad89140d9bffe95a48aaf5aaa465c1bc4d977391c239554786da1ab4" exitCode=0 Oct 30 22:16:18.379814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.379411 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" event={"ID":"b67346a3-27eb-41a9-871a-9947ad851b1a","Type":"ContainerDied","Data":"7fa71253ad89140d9bffe95a48aaf5aaa465c1bc4d977391c239554786da1ab4"} Oct 30 22:16:18.586186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.586158 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:16:18.600406 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.600382 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:16:18.719597 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:18.719568 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48074c0f_ddd9_4a6f_ac8a_c4397b7db0cd.slice/crio-22c62a1d121c0a40fd688ccdf3b1a81bf9d30fd013e7911de0d345ae815831e8 WatchSource:0}: Error finding container 22c62a1d121c0a40fd688ccdf3b1a81bf9d30fd013e7911de0d345ae815831e8: Status 404 returned error can't find the container with id 22c62a1d121c0a40fd688ccdf3b1a81bf9d30fd013e7911de0d345ae815831e8 Oct 30 22:16:18.720837 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.720816 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg"] Oct 30 22:16:18.747667 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.747633 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-tmp-dir\") pod \"570cfea8-2743-4158-8d10-07264e9b7d9f\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " Oct 30 22:16:18.748062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748039 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-home\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-home\") pod \"2a11940b-1bc9-4503-8788-00f9e8000787\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " Oct 30 22:16:18.748239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748222 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a11940b-1bc9-4503-8788-00f9e8000787-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11940b-1bc9-4503-8788-00f9e8000787-tls-certs\") pod \"2a11940b-1bc9-4503-8788-00f9e8000787\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " Oct 30 22:16:18.748449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748432 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/570cfea8-2743-4158-8d10-07264e9b7d9f-tls-certs\" (UniqueName: \"kubernetes.io/secret/570cfea8-2743-4158-8d10-07264e9b7d9f-tls-certs\") pod \"570cfea8-2743-4158-8d10-07264e9b7d9f\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " Oct 30 22:16:18.748628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748600 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-dshm\") pod \"570cfea8-2743-4158-8d10-07264e9b7d9f\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " Oct 30 22:16:18.748705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748666 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a11940b-1bc9-4503-8788-00f9e8000787-kube-api-access-hf2bq\" (UniqueName: \"kubernetes.io/projected/2a11940b-1bc9-4503-8788-00f9e8000787-kube-api-access-hf2bq\") pod \"2a11940b-1bc9-4503-8788-00f9e8000787\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " Oct 30 22:16:18.748759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748712 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-tmp-dir\") pod \"2a11940b-1bc9-4503-8788-00f9e8000787\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " Oct 30 22:16:18.748831 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748815 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570cfea8-2743-4158-8d10-07264e9b7d9f-kube-api-access-bfrrc\" (UniqueName: \"kubernetes.io/projected/570cfea8-2743-4158-8d10-07264e9b7d9f-kube-api-access-bfrrc\") pod \"570cfea8-2743-4158-8d10-07264e9b7d9f\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " Oct 30 22:16:18.748950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748928 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-home\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-home\") pod \"570cfea8-2743-4158-8d10-07264e9b7d9f\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " Oct 30 22:16:18.749120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.748988 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-dshm\") pod \"2a11940b-1bc9-4503-8788-00f9e8000787\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " Oct 30 22:16:18.749120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.749035 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-model-cache\") pod \"2a11940b-1bc9-4503-8788-00f9e8000787\" (UID: \"2a11940b-1bc9-4503-8788-00f9e8000787\") " Oct 30 22:16:18.749120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.749086 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-model-cache\") pod \"570cfea8-2743-4158-8d10-07264e9b7d9f\" (UID: \"570cfea8-2743-4158-8d10-07264e9b7d9f\") " Oct 30 22:16:18.750129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.750107 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-model-cache" pod "570cfea8-2743-4158-8d10-07264e9b7d9f" (UID: "570cfea8-2743-4158-8d10-07264e9b7d9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.750367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.750306 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-tmp-dir" pod "570cfea8-2743-4158-8d10-07264e9b7d9f" (UID: "570cfea8-2743-4158-8d10-07264e9b7d9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.750550 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.750531 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-home" pod "2a11940b-1bc9-4503-8788-00f9e8000787" (UID: "2a11940b-1bc9-4503-8788-00f9e8000787"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.751293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.751267 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-home" pod "570cfea8-2743-4158-8d10-07264e9b7d9f" (UID: "570cfea8-2743-4158-8d10-07264e9b7d9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.752071 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.752050 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-tmp-dir" pod "2a11940b-1bc9-4503-8788-00f9e8000787" (UID: "2a11940b-1bc9-4503-8788-00f9e8000787"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.757178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.757148 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570cfea8-2743-4158-8d10-07264e9b7d9f-tls-certs" pod "570cfea8-2743-4158-8d10-07264e9b7d9f" (UID: "570cfea8-2743-4158-8d10-07264e9b7d9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:18.757285 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.757175 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570cfea8-2743-4158-8d10-07264e9b7d9f-kube-api-access-bfrrc" pod "570cfea8-2743-4158-8d10-07264e9b7d9f" (UID: "570cfea8-2743-4158-8d10-07264e9b7d9f"). InnerVolumeSpecName "kube-api-access-bfrrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.757285 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.757259 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-dshm" pod "570cfea8-2743-4158-8d10-07264e9b7d9f" (UID: "570cfea8-2743-4158-8d10-07264e9b7d9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.757401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.757309 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a11940b-1bc9-4503-8788-00f9e8000787-tls-certs" pod "2a11940b-1bc9-4503-8788-00f9e8000787" (UID: "2a11940b-1bc9-4503-8788-00f9e8000787"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:18.759470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.757658 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.759470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.757801 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.759470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.757893 4933 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 22:16:18.759470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.758604 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-model-cache" pod "2a11940b-1bc9-4503-8788-00f9e8000787" (UID: "2a11940b-1bc9-4503-8788-00f9e8000787"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.760370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.759770 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a11940b-1bc9-4503-8788-00f9e8000787-kube-api-access-hf2bq" pod "2a11940b-1bc9-4503-8788-00f9e8000787" (UID: "2a11940b-1bc9-4503-8788-00f9e8000787"). InnerVolumeSpecName "kube-api-access-hf2bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:18.763543 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.761355 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-dshm" pod "2a11940b-1bc9-4503-8788-00f9e8000787" (UID: "2a11940b-1bc9-4503-8788-00f9e8000787"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:18.852473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852440 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852474 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfrrc\" (UniqueName: \"kubernetes.io/projected/570cfea8-2743-4158-8d10-07264e9b7d9f-kube-api-access-bfrrc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852512 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852526 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852538 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852550 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852562 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852573 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a11940b-1bc9-4503-8788-00f9e8000787-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852584 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a11940b-1bc9-4503-8788-00f9e8000787-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852595 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/570cfea8-2743-4158-8d10-07264e9b7d9f-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852606 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/570cfea8-2743-4158-8d10-07264e9b7d9f-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:18.852944 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:18.852619 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf2bq\" (UniqueName: \"kubernetes.io/projected/2a11940b-1bc9-4503-8788-00f9e8000787-kube-api-access-hf2bq\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:19.388636 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.388597 4933 generic.go:356] "Generic (PLEG): container finished" podID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerID="379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478" exitCode=0 Oct 30 22:16:19.389093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.388655 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" Oct 30 22:16:19.389093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.388675 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" event={"ID":"570cfea8-2743-4158-8d10-07264e9b7d9f","Type":"ContainerDied","Data":"379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478"} Oct 30 22:16:19.389093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.388719 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t" event={"ID":"570cfea8-2743-4158-8d10-07264e9b7d9f","Type":"ContainerDied","Data":"dfd6c776d89f6d74976b3b4c14b3940ff26046b12e63c4b9b7ed631ecce2cfbb"} Oct 30 22:16:19.389093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.388738 4933 scope.go:122] "RemoveContainer" containerID="379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478" Oct 30 22:16:19.390479 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.390457 4933 generic.go:356] "Generic (PLEG): container finished" podID="2a11940b-1bc9-4503-8788-00f9e8000787" containerID="9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f" exitCode=0 Oct 30 22:16:19.390640 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.390564 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" Oct 30 22:16:19.390704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.390565 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" event={"ID":"2a11940b-1bc9-4503-8788-00f9e8000787","Type":"ContainerDied","Data":"9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f"} Oct 30 22:16:19.390704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.390679 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf" event={"ID":"2a11940b-1bc9-4503-8788-00f9e8000787","Type":"ContainerDied","Data":"60cd0529829ea0a21f3c991bcff23d3356d73d26c3477b9963fccb9ad137e042"} Oct 30 22:16:19.392825 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.392801 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" event={"ID":"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd","Type":"ContainerStarted","Data":"a0a64024af02b745e325f1464c38e70f9fb56a94ff30d95bee61deebc79c35d9"} Oct 30 22:16:19.392951 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.392867 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" event={"ID":"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd","Type":"ContainerStarted","Data":"22c62a1d121c0a40fd688ccdf3b1a81bf9d30fd013e7911de0d345ae815831e8"} Oct 30 22:16:19.401707 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.401612 4933 scope.go:122] "RemoveContainer" containerID="379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478" Oct 30 22:16:19.401995 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:19.401972 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478\": container with ID starting with 379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478 not found: ID does not exist" containerID="379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478" Oct 30 22:16:19.402047 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.402008 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478"} err="failed to get container status \"379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478\": rpc error: code = NotFound desc = could not find container \"379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478\": container with ID starting with 379158a2662b64f0ca19bdaa6c47159e230fe6618be3c31949c347223c290478 not found: ID does not exist" Oct 30 22:16:19.402047 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.402031 4933 scope.go:122] "RemoveContainer" containerID="9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f" Oct 30 22:16:19.417442 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.417422 4933 scope.go:122] "RemoveContainer" containerID="9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f" Oct 30 22:16:19.417911 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:19.417881 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f\": container with ID starting with 9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f not found: ID does not exist" containerID="9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f" Oct 30 22:16:19.418001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.417922 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f"} err="failed to get container status \"9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f\": rpc error: code = NotFound desc = could not find container \"9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f\": container with ID starting with 9abd96ec2f9725753584756fd5f86c58ec4f32f74d939f6bac8ad945930ee77f not found: ID does not exist" Oct 30 22:16:19.491919 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.491318 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bc70b4-aaa3-482e-bd60-08ec61c2e18c" path="/var/lib/kubelet/pods/a0bc70b4-aaa3-482e-bd60-08ec61c2e18c/volumes" Oct 30 22:16:19.581270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.580959 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t"] Oct 30 22:16:19.588668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.588603 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/premium-w4-b0c55f-kserve-5b88cd768f-s848t"] Oct 30 22:16:19.647555 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.647432 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:19.700474 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.698580 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podStartSLOduration=2.698562028 podStartE2EDuration="2.698562028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:19.694334072 +0000 UTC m=+2028.375483978" watchObservedRunningTime="2026-10-30 22:16:19.698562028 +0000 UTC m=+2028.379711921" Oct 30 22:16:19.705873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.705811 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:19.785849 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.785808 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-xks-ca\") pod \"b67346a3-27eb-41a9-871a-9947ad851b1a\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " Oct 30 22:16:19.786048 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.785876 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-openshift-ca\") pod \"b67346a3-27eb-41a9-871a-9947ad851b1a\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " Oct 30 22:16:19.786109 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.786054 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67346a3-27eb-41a9-871a-9947ad851b1a-kube-api-access-q85jz\" (UniqueName: \"kubernetes.io/projected/b67346a3-27eb-41a9-871a-9947ad851b1a-kube-api-access-q85jz\") pod \"b67346a3-27eb-41a9-871a-9947ad851b1a\" (UID: \"b67346a3-27eb-41a9-871a-9947ad851b1a\") " Oct 30 22:16:19.786945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.786895 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-openshift-ca" pod "b67346a3-27eb-41a9-871a-9947ad851b1a" (UID: "b67346a3-27eb-41a9-871a-9947ad851b1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:19.787869 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.787841 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:19.790386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.790349 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-xks-ca" pod "b67346a3-27eb-41a9-871a-9947ad851b1a" (UID: "b67346a3-27eb-41a9-871a-9947ad851b1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:19.796799 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.796772 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67346a3-27eb-41a9-871a-9947ad851b1a-kube-api-access-q85jz" pod "b67346a3-27eb-41a9-871a-9947ad851b1a" (UID: "b67346a3-27eb-41a9-871a-9947ad851b1a"). InnerVolumeSpecName "kube-api-access-q85jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:19.869648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.869601 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf"] Oct 30 22:16:19.871419 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.871384 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b0c55f/simulator-w4-b0c55f-kserve-58c999cffb-jjzcf"] Oct 30 22:16:19.889086 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.889058 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-openshift-ca\") pod \"cfcd3373-7782-4c15-a862-12c28d631416\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " Oct 30 22:16:19.889248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.889131 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcd3373-7782-4c15-a862-12c28d631416-kube-api-access-zfljj\" (UniqueName: \"kubernetes.io/projected/cfcd3373-7782-4c15-a862-12c28d631416-kube-api-access-zfljj\") pod \"cfcd3373-7782-4c15-a862-12c28d631416\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " Oct 30 22:16:19.889319 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.889267 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-xks-ca\") pod \"cfcd3373-7782-4c15-a862-12c28d631416\" (UID: \"cfcd3373-7782-4c15-a862-12c28d631416\") " Oct 30 22:16:19.889545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.889519 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-openshift-ca" pod "cfcd3373-7782-4c15-a862-12c28d631416" (UID: "cfcd3373-7782-4c15-a862-12c28d631416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:19.889906 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.889847 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67346a3-27eb-41a9-871a-9947ad851b1a-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:19.889906 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.889876 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q85jz\" (UniqueName: \"kubernetes.io/projected/b67346a3-27eb-41a9-871a-9947ad851b1a-kube-api-access-q85jz\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:19.889906 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.889887 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:19.891504 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.891467 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-xks-ca" pod "cfcd3373-7782-4c15-a862-12c28d631416" (UID: "cfcd3373-7782-4c15-a862-12c28d631416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:19.891628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.891603 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcd3373-7782-4c15-a862-12c28d631416-kube-api-access-zfljj" pod "cfcd3373-7782-4c15-a862-12c28d631416" (UID: "cfcd3373-7782-4c15-a862-12c28d631416"). InnerVolumeSpecName "kube-api-access-zfljj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:19.990864 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.990775 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfljj\" (UniqueName: \"kubernetes.io/projected/cfcd3373-7782-4c15-a862-12c28d631416-kube-api-access-zfljj\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:19.990864 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:19.990807 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfcd3373-7782-4c15-a862-12c28d631416-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:20.260570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.260054 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:20.265512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.262648 4933 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 22:16:20.265512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.262709 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 22:16:20.401792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.401763 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" Oct 30 22:16:20.401792 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.401776 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts" event={"ID":"cfcd3373-7782-4c15-a862-12c28d631416","Type":"ContainerDied","Data":"2564df3d27960af29a2e34417477c0eb0d67a67bd851596a6b03f5b29d61921a"} Oct 30 22:16:20.402206 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.401829 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2564df3d27960af29a2e34417477c0eb0d67a67bd851596a6b03f5b29d61921a" Oct 30 22:16:20.403779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.403762 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" Oct 30 22:16:20.403927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.403841 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls" event={"ID":"b67346a3-27eb-41a9-871a-9947ad851b1a","Type":"ContainerDied","Data":"7530ed8fef34ebe65624f446b07ac6d5a4bf3670fe6d3fc9f989eadf286bb95e"} Oct 30 22:16:20.403927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:20.403877 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7530ed8fef34ebe65624f446b07ac6d5a4bf3670fe6d3fc9f989eadf286bb95e" Oct 30 22:16:21.260515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:21.260013 4933 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 22:16:21.260515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:21.260068 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 22:16:21.473300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:21.471682 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" path="/var/lib/kubelet/pods/2a11940b-1bc9-4503-8788-00f9e8000787/volumes" Oct 30 22:16:21.473300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:21.472108 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" path="/var/lib/kubelet/pods/570cfea8-2743-4158-8d10-07264e9b7d9f/volumes" Oct 30 22:16:22.267516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:22.266921 4933 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 22:16:22.267516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:22.266982 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 22:16:23.261519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:23.260032 4933 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 22:16:23.261519 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:23.260109 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 22:16:24.260514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:24.259900 4933 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 22:16:24.260514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:24.259964 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 22:16:25.263096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.263011 4933 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 22:16:25.271405 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.263106 4933 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 22:16:25.967277 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967236 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl"] Oct 30 22:16:25.967873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967854 4933 cpu_manager.go:414] "containerMap: removing container" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" Oct 30 22:16:25.967873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967871 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967890 4933 cpu_manager.go:414] "containerMap: removing container" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967896 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967909 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b67346a3-27eb-41a9-871a-9947ad851b1a" containerName="revoke-keys" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967914 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67346a3-27eb-41a9-871a-9947ad851b1a" containerName="revoke-keys" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967936 4933 cpu_manager.go:414] "containerMap: removing container" podUID="cfcd3373-7782-4c15-a862-12c28d631416" containerName="revoke-keys" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.967941 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcd3373-7782-4c15-a862-12c28d631416" containerName="revoke-keys" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.968023 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a11940b-1bc9-4503-8788-00f9e8000787" containerName="main" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.968033 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="570cfea8-2743-4158-8d10-07264e9b7d9f" containerName="main" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.968041 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67346a3-27eb-41a9-871a-9947ad851b1a" containerName="revoke-keys" Oct 30 22:16:25.968084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.968049 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcd3373-7782-4c15-a862-12c28d631416" containerName="revoke-keys" Oct 30 22:16:25.971629 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:25.971606 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.027034 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.026998 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.027246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.027226 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfnj\" (UniqueName: \"kubernetes.io/projected/c668cb8b-53e4-42fd-abc5-f5da41cb4260-kube-api-access-2qfnj\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.027397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.027376 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.028569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.028533 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl"] Oct 30 22:16:26.074223 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.074187 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:26.074223 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.074206 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:26.074223 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.074223 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:16:26.079153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.079133 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:26.128816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.128787 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.128939 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.128840 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.128985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.128970 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfnj\" (UniqueName: \"kubernetes.io/projected/c668cb8b-53e4-42fd-abc5-f5da41cb4260-kube-api-access-2qfnj\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.129545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.129523 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.131279 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.131261 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.186846 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.186824 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfnj\" (UniqueName: \"kubernetes.io/projected/c668cb8b-53e4-42fd-abc5-f5da41cb4260-kube-api-access-2qfnj\") pod \"maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.264644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.264584 4933 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:26.265002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.264980 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:26.265656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.265640 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:16:26.285821 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.285798 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:26.677372 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:26.677330 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc668cb8b_53e4_42fd_abc5_f5da41cb4260.slice/crio-3fca1b14d79dd0fd00903a9acd24912f4a6f9c1df96bc52585416d3cbe5dd367 WatchSource:0}: Error finding container 3fca1b14d79dd0fd00903a9acd24912f4a6f9c1df96bc52585416d3cbe5dd367: Status 404 returned error can't find the container with id 3fca1b14d79dd0fd00903a9acd24912f4a6f9c1df96bc52585416d3cbe5dd367 Oct 30 22:16:26.678991 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:26.678966 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl"] Oct 30 22:16:27.568322 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:27.568296 4933 generic.go:356] "Generic (PLEG): container finished" podID="c668cb8b-53e4-42fd-abc5-f5da41cb4260" containerID="14927c04e9990508cd70078b58ef9feaeab0434939a7c154aceff3176bce6739" exitCode=0 Oct 30 22:16:27.568709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:27.568378 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" event={"ID":"c668cb8b-53e4-42fd-abc5-f5da41cb4260","Type":"ContainerDied","Data":"14927c04e9990508cd70078b58ef9feaeab0434939a7c154aceff3176bce6739"} Oct 30 22:16:27.568709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:27.568443 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" event={"ID":"c668cb8b-53e4-42fd-abc5-f5da41cb4260","Type":"ContainerStarted","Data":"3fca1b14d79dd0fd00903a9acd24912f4a6f9c1df96bc52585416d3cbe5dd367"} Oct 30 22:16:28.572725 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:28.572684 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:16:28.733564 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.733540 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:28.776553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.776524 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c668cb8b-53e4-42fd-abc5-f5da41cb4260-kube-api-access-2qfnj\" (UniqueName: \"kubernetes.io/projected/c668cb8b-53e4-42fd-abc5-f5da41cb4260-kube-api-access-2qfnj\") pod \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " Oct 30 22:16:28.776770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.776754 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-xks-ca\") pod \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " Oct 30 22:16:28.776825 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.776812 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-openshift-ca\") pod \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\" (UID: \"c668cb8b-53e4-42fd-abc5-f5da41cb4260\") " Oct 30 22:16:28.777182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.777155 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-openshift-ca" pod "c668cb8b-53e4-42fd-abc5-f5da41cb4260" (UID: "c668cb8b-53e4-42fd-abc5-f5da41cb4260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:28.778785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.778760 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c668cb8b-53e4-42fd-abc5-f5da41cb4260-kube-api-access-2qfnj" pod "c668cb8b-53e4-42fd-abc5-f5da41cb4260" (UID: "c668cb8b-53e4-42fd-abc5-f5da41cb4260"). InnerVolumeSpecName "kube-api-access-2qfnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:28.778873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.778806 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-xks-ca" pod "c668cb8b-53e4-42fd-abc5-f5da41cb4260" (UID: "c668cb8b-53e4-42fd-abc5-f5da41cb4260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:28.881278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.881134 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:28.881567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.881547 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c668cb8b-53e4-42fd-abc5-f5da41cb4260-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:28.881754 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:28.881739 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qfnj\" (UniqueName: \"kubernetes.io/projected/c668cb8b-53e4-42fd-abc5-f5da41cb4260-kube-api-access-2qfnj\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:29.207620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.207548 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq"] Oct 30 22:16:29.208095 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.208083 4933 cpu_manager.go:414] "containerMap: removing container" podUID="c668cb8b-53e4-42fd-abc5-f5da41cb4260" containerName="revoke-keys" Oct 30 22:16:29.208138 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.208097 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c668cb8b-53e4-42fd-abc5-f5da41cb4260" containerName="revoke-keys" Oct 30 22:16:29.208217 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.208209 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="c668cb8b-53e4-42fd-abc5-f5da41cb4260" containerName="revoke-keys" Oct 30 22:16:29.211645 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.211628 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.269745 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.269719 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq"] Oct 30 22:16:29.389483 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.389454 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.389670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.389575 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flbfg\" (UniqueName: \"kubernetes.io/projected/ae3062a4-83a4-4198-928f-6dbc25fad9e0-kube-api-access-flbfg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.389750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.389733 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.491579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.491546 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.491747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.491614 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flbfg\" (UniqueName: \"kubernetes.io/projected/ae3062a4-83a4-4198-928f-6dbc25fad9e0-kube-api-access-flbfg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.491830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.491760 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.492146 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.492126 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.494333 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.494309 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.549370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.549345 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flbfg\" (UniqueName: \"kubernetes.io/projected/ae3062a4-83a4-4198-928f-6dbc25fad9e0-kube-api-access-flbfg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:29.590370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.590349 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" Oct 30 22:16:29.590712 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.590366 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl" event={"ID":"c668cb8b-53e4-42fd-abc5-f5da41cb4260","Type":"ContainerDied","Data":"3fca1b14d79dd0fd00903a9acd24912f4a6f9c1df96bc52585416d3cbe5dd367"} Oct 30 22:16:29.590712 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.590421 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fca1b14d79dd0fd00903a9acd24912f4a6f9c1df96bc52585416d3cbe5dd367" Oct 30 22:16:29.825649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:29.825622 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:30.212527 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:30.212464 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3062a4_83a4_4198_928f_6dbc25fad9e0.slice/crio-75a85814d8dc072bf19274674bc846d42e58e21e55c9e793c029aeca24db053c WatchSource:0}: Error finding container 75a85814d8dc072bf19274674bc846d42e58e21e55c9e793c029aeca24db053c: Status 404 returned error can't find the container with id 75a85814d8dc072bf19274674bc846d42e58e21e55c9e793c029aeca24db053c Oct 30 22:16:30.214956 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.214933 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq"] Oct 30 22:16:30.266165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.266139 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs"] Oct 30 22:16:30.266399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.266366 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" containerID="cri-o://f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4" gracePeriod=30 Oct 30 22:16:30.383432 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.383399 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t"] Oct 30 22:16:30.388725 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.388701 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5d36f6-5fw4t"] Oct 30 22:16:30.448367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.448336 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4"] Oct 30 22:16:30.461232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.461169 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh"] Oct 30 22:16:30.599889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.599858 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" event={"ID":"ae3062a4-83a4-4198-928f-6dbc25fad9e0","Type":"ContainerStarted","Data":"835ce065944f4bf69f5425d2a898c7d550a4d7418dd7d693fa9706bd441bc389"} Oct 30 22:16:30.600329 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.599910 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" event={"ID":"ae3062a4-83a4-4198-928f-6dbc25fad9e0","Type":"ContainerStarted","Data":"75a85814d8dc072bf19274674bc846d42e58e21e55c9e793c029aeca24db053c"} Oct 30 22:16:30.799136 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:30.799091 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" podStartSLOduration=1.799078539 podStartE2EDuration="1.799078539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:30.798473558 +0000 UTC m=+2039.479623452" watchObservedRunningTime="2026-10-30 22:16:30.799078539 +0000 UTC m=+2039.480228431" Oct 30 22:16:31.468558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:31.468469 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883ce997-120f-4c76-97fa-b60eb5e81b71" path="/var/lib/kubelet/pods/883ce997-120f-4c76-97fa-b60eb5e81b71/volumes" Oct 30 22:16:31.610972 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:31.610940 4933 generic.go:356] "Generic (PLEG): container finished" podID="ae3062a4-83a4-4198-928f-6dbc25fad9e0" containerID="835ce065944f4bf69f5425d2a898c7d550a4d7418dd7d693fa9706bd441bc389" exitCode=0 Oct 30 22:16:31.611400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:31.611031 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" event={"ID":"ae3062a4-83a4-4198-928f-6dbc25fad9e0","Type":"ContainerDied","Data":"835ce065944f4bf69f5425d2a898c7d550a4d7418dd7d693fa9706bd441bc389"} Oct 30 22:16:32.821942 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.821906 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:32.931269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.931197 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl"] Oct 30 22:16:32.931424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.931387 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" podUID="50246925-1aa8-4563-a2eb-47a1cd64c692" containerName="discovery" containerID="cri-o://5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e" gracePeriod=30 Oct 30 22:16:32.936293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.936265 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3062a4-83a4-4198-928f-6dbc25fad9e0-kube-api-access-flbfg\" (UniqueName: \"kubernetes.io/projected/ae3062a4-83a4-4198-928f-6dbc25fad9e0-kube-api-access-flbfg\") pod \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " Oct 30 22:16:32.936473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.936457 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-xks-ca\") pod \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " Oct 30 22:16:32.936648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.936516 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-openshift-ca\") pod \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\" (UID: \"ae3062a4-83a4-4198-928f-6dbc25fad9e0\") " Oct 30 22:16:32.936898 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.936870 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-openshift-ca" pod "ae3062a4-83a4-4198-928f-6dbc25fad9e0" (UID: "ae3062a4-83a4-4198-928f-6dbc25fad9e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:32.937108 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.937092 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:32.938710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.938682 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3062a4-83a4-4198-928f-6dbc25fad9e0-kube-api-access-flbfg" pod "ae3062a4-83a4-4198-928f-6dbc25fad9e0" (UID: "ae3062a4-83a4-4198-928f-6dbc25fad9e0"). InnerVolumeSpecName "kube-api-access-flbfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:32.938814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:32.938798 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-xks-ca" pod "ae3062a4-83a4-4198-928f-6dbc25fad9e0" (UID: "ae3062a4-83a4-4198-928f-6dbc25fad9e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:33.037967 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.037934 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae3062a4-83a4-4198-928f-6dbc25fad9e0-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.037967 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.037963 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flbfg\" (UniqueName: \"kubernetes.io/projected/ae3062a4-83a4-4198-928f-6dbc25fad9e0-kube-api-access-flbfg\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.203952 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.203926 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:16:33.240299 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.240272 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-kubeconfig\") pod \"50246925-1aa8-4563-a2eb-47a1cd64c692\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " Oct 30 22:16:33.240477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.240321 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-token\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-token\") pod \"50246925-1aa8-4563-a2eb-47a1cd64c692\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " Oct 30 22:16:33.240477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.240416 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-cacerts\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-cacerts\") pod \"50246925-1aa8-4563-a2eb-47a1cd64c692\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " Oct 30 22:16:33.240477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.240462 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50246925-1aa8-4563-a2eb-47a1cd64c692-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50246925-1aa8-4563-a2eb-47a1cd64c692-local-certs\") pod \"50246925-1aa8-4563-a2eb-47a1cd64c692\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " Oct 30 22:16:33.240718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.240605 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-ca-configmap\") pod \"50246925-1aa8-4563-a2eb-47a1cd64c692\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " Oct 30 22:16:33.240718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.240650 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-dns-cert\") pod \"50246925-1aa8-4563-a2eb-47a1cd64c692\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " Oct 30 22:16:33.240843 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.240777 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-kube-api-access-85ztl\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-kube-api-access-85ztl\") pod \"50246925-1aa8-4563-a2eb-47a1cd64c692\" (UID: \"50246925-1aa8-4563-a2eb-47a1cd64c692\") " Oct 30 22:16:33.241362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.241299 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-ca-configmap" pod "50246925-1aa8-4563-a2eb-47a1cd64c692" (UID: "50246925-1aa8-4563-a2eb-47a1cd64c692"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:33.241923 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.241897 4933 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-ca-configmap\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.243362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.243330 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-kubeconfig" pod "50246925-1aa8-4563-a2eb-47a1cd64c692" (UID: "50246925-1aa8-4563-a2eb-47a1cd64c692"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:33.243466 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.243358 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50246925-1aa8-4563-a2eb-47a1cd64c692-local-certs" pod "50246925-1aa8-4563-a2eb-47a1cd64c692" (UID: "50246925-1aa8-4563-a2eb-47a1cd64c692"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:33.243466 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.243384 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-cacerts" pod "50246925-1aa8-4563-a2eb-47a1cd64c692" (UID: "50246925-1aa8-4563-a2eb-47a1cd64c692"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:33.243674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.243616 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-token" pod "50246925-1aa8-4563-a2eb-47a1cd64c692" (UID: "50246925-1aa8-4563-a2eb-47a1cd64c692"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:33.243738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.243716 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-kube-api-access-85ztl" pod "50246925-1aa8-4563-a2eb-47a1cd64c692" (UID: "50246925-1aa8-4563-a2eb-47a1cd64c692"). InnerVolumeSpecName "kube-api-access-85ztl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:33.244101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.244080 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-dns-cert" pod "50246925-1aa8-4563-a2eb-47a1cd64c692" (UID: "50246925-1aa8-4563-a2eb-47a1cd64c692"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:33.343634 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.343603 4933 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-csr-dns-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.343634 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.343634 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85ztl\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-kube-api-access-85ztl\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.343866 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.343645 4933 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-kubeconfig\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.343866 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.343654 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50246925-1aa8-4563-a2eb-47a1cd64c692-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.343866 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.343664 4933 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50246925-1aa8-4563-a2eb-47a1cd64c692-cacerts\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.343866 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.343672 4933 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50246925-1aa8-4563-a2eb-47a1cd64c692-local-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:33.628841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.628809 4933 generic.go:356] "Generic (PLEG): container finished" podID="50246925-1aa8-4563-a2eb-47a1cd64c692" containerID="5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e" exitCode=0 Oct 30 22:16:33.629009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.628865 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" Oct 30 22:16:33.629009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.628891 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" event={"ID":"50246925-1aa8-4563-a2eb-47a1cd64c692","Type":"ContainerDied","Data":"5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e"} Oct 30 22:16:33.629009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.628948 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl" event={"ID":"50246925-1aa8-4563-a2eb-47a1cd64c692","Type":"ContainerDied","Data":"7ef1da8b3b763279a3b3048ddd05abf1d1bb1a17b389dd463d7264facff3dd80"} Oct 30 22:16:33.629009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.628975 4933 scope.go:122] "RemoveContainer" containerID="5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e" Oct 30 22:16:33.630801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.630734 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" event={"ID":"ae3062a4-83a4-4198-928f-6dbc25fad9e0","Type":"ContainerDied","Data":"75a85814d8dc072bf19274674bc846d42e58e21e55c9e793c029aeca24db053c"} Oct 30 22:16:33.630801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.630776 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a85814d8dc072bf19274674bc846d42e58e21e55c9e793c029aeca24db053c" Oct 30 22:16:33.630912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.630834 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq" Oct 30 22:16:33.642639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.642619 4933 scope.go:122] "RemoveContainer" containerID="5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e" Oct 30 22:16:33.642985 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:33.642963 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e\": container with ID starting with 5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e not found: ID does not exist" containerID="5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e" Oct 30 22:16:33.643059 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.642997 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e"} err="failed to get container status \"5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e\": rpc error: code = NotFound desc = could not find container \"5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e\": container with ID starting with 5be059abb5c466e683fdc586cbe548d44e25a4e1700ef92d862cd55cb8e2654e not found: ID does not exist" Oct 30 22:16:33.849861 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.849837 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl"] Oct 30 22:16:33.853668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:33.853647 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9z7tl"] Oct 30 22:16:35.478522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.478472 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50246925-1aa8-4563-a2eb-47a1cd64c692" path="/var/lib/kubelet/pods/50246925-1aa8-4563-a2eb-47a1cd64c692/volumes" Oct 30 22:16:35.551344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.551285 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:16:35.568760 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.568738 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-kube-api-access-q8hwh\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-kube-api-access-q8hwh\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.568896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.568815 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-socket\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.568896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.568849 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-podinfo\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.569013 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.568915 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-envoy\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.569013 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.568970 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-credential-socket\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.569127 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569021 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-token\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-token\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.569127 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569069 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-data\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.569229 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569134 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-certs\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.569229 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569148 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-socket" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:35.569229 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569192 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istiod-ca-cert\") pod \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\" (UID: \"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97\") " Oct 30 22:16:35.569590 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569563 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-data" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:35.569788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569769 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.570029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569793 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.570029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569854 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istiod-ca-cert" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:35.570029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569849 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-credential-socket" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:35.570029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.569699 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-certs" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:35.571674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.571627 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-kube-api-access-q8hwh" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "kube-api-access-q8hwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:35.571674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.571664 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-token" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:35.572002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.571796 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-podinfo" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:35.572002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.571967 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-envoy" pod "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" (UID: "bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:35.649469 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.649440 4933 generic.go:356] "Generic (PLEG): container finished" podID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerID="f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4" exitCode=0 Oct 30 22:16:35.649595 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.649516 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" Oct 30 22:16:35.649635 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.649478 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" event={"ID":"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97","Type":"ContainerDied","Data":"f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4"} Oct 30 22:16:35.649681 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.649649 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs" event={"ID":"bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97","Type":"ContainerDied","Data":"b6b9f0b69358eca0725a1003ac8b2dd5bcf5b447184e39460f5fd0ad96cb2e22"} Oct 30 22:16:35.649713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.649680 4933 scope.go:122] "RemoveContainer" containerID="f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4" Oct 30 22:16:35.670941 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.670912 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.671062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.670945 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.671062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.670958 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.671062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.670970 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.671062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.670982 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.671062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.670994 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.671062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.671007 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8hwh\" (UniqueName: \"kubernetes.io/projected/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97-kube-api-access-q8hwh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:35.672686 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.672662 4933 scope.go:122] "RemoveContainer" containerID="f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4" Oct 30 22:16:35.673018 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:35.672999 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4\": container with ID starting with f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4 not found: ID does not exist" containerID="f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4" Oct 30 22:16:35.673101 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.673025 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4"} err="failed to get container status \"f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4\": rpc error: code = NotFound desc = could not find container \"f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4\": container with ID starting with f9ef2caf390db4e298328ecbf351a40998e2b41f82dd1ba37fae96a24ad9f2b4 not found: ID does not exist" Oct 30 22:16:35.919626 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.919541 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs"] Oct 30 22:16:35.923530 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:35.923484 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-c2vvs"] Oct 30 22:16:37.466863 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:37.466834 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" path="/var/lib/kubelet/pods/bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97/volumes" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.392447 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9"] Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393095 4933 cpu_manager.go:414] "containerMap: removing container" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393110 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393134 4933 cpu_manager.go:414] "containerMap: removing container" podUID="50246925-1aa8-4563-a2eb-47a1cd64c692" containerName="discovery" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393143 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50246925-1aa8-4563-a2eb-47a1cd64c692" containerName="discovery" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393153 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ae3062a4-83a4-4198-928f-6dbc25fad9e0" containerName="revoke-keys" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393159 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3062a4-83a4-4198-928f-6dbc25fad9e0" containerName="revoke-keys" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393281 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbcc83d-9e4a-4dda-9936-7b0b1f3bbe97" containerName="istio-proxy" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393300 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="50246925-1aa8-4563-a2eb-47a1cd64c692" containerName="discovery" Oct 30 22:16:39.401521 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.393307 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3062a4-83a4-4198-928f-6dbc25fad9e0" containerName="revoke-keys" Oct 30 22:16:39.402621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.401908 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.494545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.494392 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.494785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.494613 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8z7\" (UniqueName: \"kubernetes.io/projected/ddba5b4d-7abb-4e2f-af97-77857ac38088-kube-api-access-rs8z7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.494785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.494666 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.529516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.528807 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:39.529516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.528937 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:16:39.529516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.529041 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:39.529516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.529155 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:39.597407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.595834 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.597407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.595944 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8z7\" (UniqueName: \"kubernetes.io/projected/ddba5b4d-7abb-4e2f-af97-77857ac38088-kube-api-access-rs8z7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.597407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.595977 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.597407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.596411 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9"] Oct 30 22:16:39.597407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.597250 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.600085 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.600026 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.655664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.655586 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8z7\" (UniqueName: \"kubernetes.io/projected/ddba5b4d-7abb-4e2f-af97-77857ac38088-kube-api-access-rs8z7\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:39.807446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:39.807393 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:40.472516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:40.469414 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9"] Oct 30 22:16:40.497636 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:40.497573 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddba5b4d_7abb_4e2f_af97_77857ac38088.slice/crio-dcbb02e9683b931991ce0ec17f7360ea2ee186753ecd48c942a769d6bab232f2 WatchSource:0}: Error finding container dcbb02e9683b931991ce0ec17f7360ea2ee186753ecd48c942a769d6bab232f2: Status 404 returned error can't find the container with id dcbb02e9683b931991ce0ec17f7360ea2ee186753ecd48c942a769d6bab232f2 Oct 30 22:16:40.532365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:40.532326 4933 scope.go:122] "RemoveContainer" containerID="245e6e84da6736cd22110c209b46b0847120bd92f2325029bcf3bb4e9aecdf3f" Oct 30 22:16:40.546990 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:40.546969 4933 scope.go:122] "RemoveContainer" containerID="dd7d2b21e747a033035fdd23e2131ab61c0ee9331ef2b45ca4134560a5224a9f" Oct 30 22:16:40.713179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:40.713144 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" event={"ID":"ddba5b4d-7abb-4e2f-af97-77857ac38088","Type":"ContainerStarted","Data":"dcbb02e9683b931991ce0ec17f7360ea2ee186753ecd48c942a769d6bab232f2"} Oct 30 22:16:41.722219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:41.722190 4933 generic.go:356] "Generic (PLEG): container finished" podID="ddba5b4d-7abb-4e2f-af97-77857ac38088" containerID="5570f8988f4fc443b40dc79d8f875e2468ded5f43331294a94f4d972a84b8606" exitCode=0 Oct 30 22:16:41.722649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:41.722243 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" event={"ID":"ddba5b4d-7abb-4e2f-af97-77857ac38088","Type":"ContainerDied","Data":"5570f8988f4fc443b40dc79d8f875e2468ded5f43331294a94f4d972a84b8606"} Oct 30 22:16:42.909974 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.909948 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:42.964937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.964915 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddba5b4d-7abb-4e2f-af97-77857ac38088-kube-api-access-rs8z7\" (UniqueName: \"kubernetes.io/projected/ddba5b4d-7abb-4e2f-af97-77857ac38088-kube-api-access-rs8z7\") pod \"ddba5b4d-7abb-4e2f-af97-77857ac38088\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " Oct 30 22:16:42.965060 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.964999 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-xks-ca\") pod \"ddba5b4d-7abb-4e2f-af97-77857ac38088\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " Oct 30 22:16:42.965102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.965073 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-openshift-ca\") pod \"ddba5b4d-7abb-4e2f-af97-77857ac38088\" (UID: \"ddba5b4d-7abb-4e2f-af97-77857ac38088\") " Oct 30 22:16:42.965438 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.965414 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-openshift-ca" pod "ddba5b4d-7abb-4e2f-af97-77857ac38088" (UID: "ddba5b4d-7abb-4e2f-af97-77857ac38088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:42.965718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.965653 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:42.967186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.967162 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-xks-ca" pod "ddba5b4d-7abb-4e2f-af97-77857ac38088" (UID: "ddba5b4d-7abb-4e2f-af97-77857ac38088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:42.967186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:42.967171 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddba5b4d-7abb-4e2f-af97-77857ac38088-kube-api-access-rs8z7" pod "ddba5b4d-7abb-4e2f-af97-77857ac38088" (UID: "ddba5b4d-7abb-4e2f-af97-77857ac38088"). InnerVolumeSpecName "kube-api-access-rs8z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:43.067373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:43.067342 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs8z7\" (UniqueName: \"kubernetes.io/projected/ddba5b4d-7abb-4e2f-af97-77857ac38088-kube-api-access-rs8z7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.067373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:43.067372 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddba5b4d-7abb-4e2f-af97-77857ac38088-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:43.738795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:43.738703 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" Oct 30 22:16:43.738795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:43.738718 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9" event={"ID":"ddba5b4d-7abb-4e2f-af97-77857ac38088","Type":"ContainerDied","Data":"dcbb02e9683b931991ce0ec17f7360ea2ee186753ecd48c942a769d6bab232f2"} Oct 30 22:16:43.738795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:43.738771 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcbb02e9683b931991ce0ec17f7360ea2ee186753ecd48c942a769d6bab232f2" Oct 30 22:16:45.494513 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.491730 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" containerID="cri-o://a0ba5eab8fc0aad73435234b52b2f9d252f8141ff84a49b2bb4731cbd04eff40" gracePeriod=45 Oct 30 22:16:45.501237 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.501199 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" containerID="cri-o://abf09d5425c89b81aeb1e39a4d580c4d53effc6095cbf6242773ad9399f409d2" gracePeriod=45 Oct 30 22:16:45.636957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.636789 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb"] Oct 30 22:16:45.661515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.637162 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" containerID="cri-o://0f1c6ba2ec315b6ba3c265490bf523b2abf26065c6409f87e16b0b7854e12484" gracePeriod=30 Oct 30 22:16:45.703715 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:45.703243 4933 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83ea6f0f_a024_44c5_b341_a96cd0b4aa24.slice/crio-abf09d5425c89b81aeb1e39a4d580c4d53effc6095cbf6242773ad9399f409d2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:16:45.707092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.707066 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x"] Oct 30 22:16:45.707344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.707287 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" containerID="cri-o://962fb143d1a66dea0b8186d64a586ed7c131ac6d6dc264af48bbb0b52894ce12" gracePeriod=30 Oct 30 22:16:45.727159 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.727126 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused" Oct 30 22:16:45.778956 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.778663 4933 generic.go:356] "Generic (PLEG): container finished" podID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerID="abf09d5425c89b81aeb1e39a4d580c4d53effc6095cbf6242773ad9399f409d2" exitCode=0 Oct 30 22:16:45.778956 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.778908 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" event={"ID":"83ea6f0f-a024-44c5-b341-a96cd0b4aa24","Type":"ContainerDied","Data":"abf09d5425c89b81aeb1e39a4d580c4d53effc6095cbf6242773ad9399f409d2"} Oct 30 22:16:45.782116 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.782079 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-6a2244"] Oct 30 22:16:45.782750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.782610 4933 generic.go:356] "Generic (PLEG): container finished" podID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerID="a0ba5eab8fc0aad73435234b52b2f9d252f8141ff84a49b2bb4731cbd04eff40" exitCode=0 Oct 30 22:16:45.782958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.782934 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ddba5b4d-7abb-4e2f-af97-77857ac38088" containerName="revoke-keys" Oct 30 22:16:45.783032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.782959 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddba5b4d-7abb-4e2f-af97-77857ac38088" containerName="revoke-keys" Oct 30 22:16:45.783118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.783105 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddba5b4d-7abb-4e2f-af97-77857ac38088" containerName="revoke-keys" Oct 30 22:16:45.787029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.787007 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" event={"ID":"7bc0a351-9f35-447d-8f82-37a2f1d42dc4","Type":"ContainerDied","Data":"a0ba5eab8fc0aad73435234b52b2f9d252f8141ff84a49b2bb4731cbd04eff40"} Oct 30 22:16:45.787129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.787099 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:45.845532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.841058 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-6a2244"] Oct 30 22:16:45.845532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.842163 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqnmc\" (UniqueName: \"kubernetes.io/projected/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7-kube-api-access-dqnmc\") pod \"test-curl-7320-6a2244\" (UID: \"aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7\") " pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:45.882397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.882342 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:16:45.902627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.902604 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:16:45.919313 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.919290 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmjfd\"" Oct 30 22:16:45.943132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943112 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-dshm\") pod \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " Oct 30 22:16:45.943243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943150 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tmp-dir\") pod \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " Oct 30 22:16:45.943243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943177 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-home\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-home\") pod \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " Oct 30 22:16:45.943243 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943209 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-model-cache\") pod \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " Oct 30 22:16:45.943472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943263 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-kube-api-access-fm986\" (UniqueName: \"kubernetes.io/projected/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-kube-api-access-fm986\") pod \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " Oct 30 22:16:45.943472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943325 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tmp-dir\") pod \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " Oct 30 22:16:45.943472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943370 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-kube-api-access-dlhfp\" (UniqueName: \"kubernetes.io/projected/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-kube-api-access-dlhfp\") pod \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " Oct 30 22:16:45.943472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943398 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tls-certs\") pod \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " Oct 30 22:16:45.943472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943464 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-home\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-home\") pod \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " Oct 30 22:16:45.943777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943536 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-dshm\") pod \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\" (UID: \"7bc0a351-9f35-447d-8f82-37a2f1d42dc4\") " Oct 30 22:16:45.943777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943603 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-model-cache\") pod \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " Oct 30 22:16:45.943777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943669 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tls-certs\" (UniqueName: \"kubernetes.io/secret/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tls-certs\") pod \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\" (UID: \"83ea6f0f-a024-44c5-b341-a96cd0b4aa24\") " Oct 30 22:16:45.943926 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943459 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-home" pod "7bc0a351-9f35-447d-8f82-37a2f1d42dc4" (UID: "7bc0a351-9f35-447d-8f82-37a2f1d42dc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.944134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.944110 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqnmc\" (UniqueName: \"kubernetes.io/projected/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7-kube-api-access-dqnmc\") pod \"test-curl-7320-6a2244\" (UID: \"aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7\") " pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:45.944518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.944451 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-model-cache" pod "83ea6f0f-a024-44c5-b341-a96cd0b4aa24" (UID: "83ea6f0f-a024-44c5-b341-a96cd0b4aa24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.944518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943474 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tmp-dir" pod "7bc0a351-9f35-447d-8f82-37a2f1d42dc4" (UID: "7bc0a351-9f35-447d-8f82-37a2f1d42dc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.944518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943822 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tmp-dir" pod "83ea6f0f-a024-44c5-b341-a96cd0b4aa24" (UID: "83ea6f0f-a024-44c5-b341-a96cd0b4aa24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.944518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.943829 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-model-cache" pod "7bc0a351-9f35-447d-8f82-37a2f1d42dc4" (UID: "7bc0a351-9f35-447d-8f82-37a2f1d42dc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.944950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.944902 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-home" pod "83ea6f0f-a024-44c5-b341-a96cd0b4aa24" (UID: "83ea6f0f-a024-44c5-b341-a96cd0b4aa24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.945087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.945074 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:45.946410 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.946384 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-kube-api-access-dlhfp" pod "83ea6f0f-a024-44c5-b341-a96cd0b4aa24" (UID: "83ea6f0f-a024-44c5-b341-a96cd0b4aa24"). InnerVolumeSpecName "kube-api-access-dlhfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:45.946529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.946473 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tls-certs" pod "7bc0a351-9f35-447d-8f82-37a2f1d42dc4" (UID: "7bc0a351-9f35-447d-8f82-37a2f1d42dc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:45.946620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.946600 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tls-certs" pod "83ea6f0f-a024-44c5-b341-a96cd0b4aa24" (UID: "83ea6f0f-a024-44c5-b341-a96cd0b4aa24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:16:45.947263 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.947233 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-dshm" pod "7bc0a351-9f35-447d-8f82-37a2f1d42dc4" (UID: "7bc0a351-9f35-447d-8f82-37a2f1d42dc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.947333 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.947310 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-dshm" pod "83ea6f0f-a024-44c5-b341-a96cd0b4aa24" (UID: "83ea6f0f-a024-44c5-b341-a96cd0b4aa24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:45.947679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:45.947661 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-kube-api-access-fm986" pod "7bc0a351-9f35-447d-8f82-37a2f1d42dc4" (UID: "7bc0a351-9f35-447d-8f82-37a2f1d42dc4"). InnerVolumeSpecName "kube-api-access-fm986". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:46.004023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.003996 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqnmc\" (UniqueName: \"kubernetes.io/projected/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7-kube-api-access-dqnmc\") pod \"test-curl-7320-6a2244\" (UID: \"aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7\") " pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:46.046323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046251 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046280 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046295 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046307 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046323 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046319 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046334 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm986\" (UniqueName: \"kubernetes.io/projected/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-kube-api-access-fm986\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046348 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046360 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhfp\" (UniqueName: \"kubernetes.io/projected/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-kube-api-access-dlhfp\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046371 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046382 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ea6f0f-a024-44c5-b341-a96cd0b4aa24-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.046663 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.046394 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bc0a351-9f35-447d-8f82-37a2f1d42dc4-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:46.110656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.110629 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:46.500150 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:46.500123 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeeb5b3c_cb65_4f56_9557_9e4179d2d4c7.slice/crio-fbf26538323a21c670ad004c126c26b28ce5af14f191aebaec4c1c1f82b4dc2d WatchSource:0}: Error finding container fbf26538323a21c670ad004c126c26b28ce5af14f191aebaec4c1c1f82b4dc2d: Status 404 returned error can't find the container with id fbf26538323a21c670ad004c126c26b28ce5af14f191aebaec4c1c1f82b4dc2d Oct 30 22:16:46.500903 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.500785 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-6a2244"] Oct 30 22:16:46.791009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.790981 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" Oct 30 22:16:46.791185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.791021 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh" event={"ID":"7bc0a351-9f35-447d-8f82-37a2f1d42dc4","Type":"ContainerDied","Data":"55c0cf3af8967c2da9c374b182f5a1a15797ce9a40fb6c0b3d0e96d705fe6eea"} Oct 30 22:16:46.791185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.791073 4933 scope.go:122] "RemoveContainer" containerID="a0ba5eab8fc0aad73435234b52b2f9d252f8141ff84a49b2bb4731cbd04eff40" Oct 30 22:16:46.792999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.792976 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-6a2244" event={"ID":"aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7","Type":"ContainerStarted","Data":"0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f"} Oct 30 22:16:46.793169 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.793148 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-6a2244" event={"ID":"aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7","Type":"ContainerStarted","Data":"fbf26538323a21c670ad004c126c26b28ce5af14f191aebaec4c1c1f82b4dc2d"} Oct 30 22:16:46.794444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.794423 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" event={"ID":"83ea6f0f-a024-44c5-b341-a96cd0b4aa24","Type":"ContainerDied","Data":"fe43d1532a8671f74e8447969d453d80c0536a402706f1c9db47c012c67e66b6"} Oct 30 22:16:46.794573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.794460 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4" Oct 30 22:16:46.802988 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.802968 4933 scope.go:122] "RemoveContainer" containerID="abf09d5425c89b81aeb1e39a4d580c4d53effc6095cbf6242773ad9399f409d2" Oct 30 22:16:46.960447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:46.960390 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7320-6a2244" podStartSLOduration=1.960377348 podStartE2EDuration="1.960377348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:46.957056668 +0000 UTC m=+2055.638206563" watchObservedRunningTime="2026-10-30 22:16:46.960377348 +0000 UTC m=+2055.641527241" Oct 30 22:16:47.066265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.066175 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8"] Oct 30 22:16:47.066481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.066456 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" containerID="cri-o://d4f5ec9a84413291bb4bccd02755d49f980b41780fac98ad6495f0878373fb05" gracePeriod=30 Oct 30 22:16:47.081034 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.080808 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh"] Oct 30 22:16:47.097118 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.097084 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/simulator-w2-0c99b1-kserve-7585dc95cf-9rsmh"] Oct 30 22:16:47.214193 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.214163 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4"] Oct 30 22:16:47.219326 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.219304 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0c99b1/premium-w2-0c99b1-kserve-867d497754-vzkv4"] Oct 30 22:16:47.432620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.432539 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:16:47.432838 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.432812 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:16:47.468181 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.468147 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" path="/var/lib/kubelet/pods/7bc0a351-9f35-447d-8f82-37a2f1d42dc4/volumes" Oct 30 22:16:47.468827 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.468800 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" path="/var/lib/kubelet/pods/83ea6f0f-a024-44c5-b341-a96cd0b4aa24/volumes" Oct 30 22:16:47.468958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.468807 4933 patch_prober.go:29] interesting pod/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:16:47.468958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:47.468868 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:16:48.416736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.416704 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-6a2244"] Oct 30 22:16:48.422932 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.422902 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-6a2244"] Oct 30 22:16:48.575765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.575734 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-18cefc"] Oct 30 22:16:48.576313 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576298 4933 cpu_manager.go:414] "containerMap: removing container" podUID="aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" containerName="test-curl-7320-6a2244" Oct 30 22:16:48.576313 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576313 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" containerName="test-curl-7320-6a2244" Oct 30 22:16:48.576464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576329 4933 cpu_manager.go:414] "containerMap: removing container" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" Oct 30 22:16:48.576464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576335 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" Oct 30 22:16:48.576464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576370 4933 cpu_manager.go:414] "containerMap: removing container" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" Oct 30 22:16:48.576464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576375 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" Oct 30 22:16:48.576464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576462 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" containerName="test-curl-7320-6a2244" Oct 30 22:16:48.576759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576473 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ea6f0f-a024-44c5-b341-a96cd0b4aa24" containerName="main" Oct 30 22:16:48.576759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.576482 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc0a351-9f35-447d-8f82-37a2f1d42dc4" containerName="main" Oct 30 22:16:48.581279 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.581261 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:48.634860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.634836 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-18cefc"] Oct 30 22:16:48.682915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.682851 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psffw\" (UniqueName: \"kubernetes.io/projected/2ddd25b6-6227-470a-8d3d-c1748e15c2f1-kube-api-access-psffw\") pod \"test-curl-7320-18cefc\" (UID: \"2ddd25b6-6227-470a-8d3d-c1748e15c2f1\") " pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:48.784465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.784435 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psffw\" (UniqueName: \"kubernetes.io/projected/2ddd25b6-6227-470a-8d3d-c1748e15c2f1-kube-api-access-psffw\") pod \"test-curl-7320-18cefc\" (UID: \"2ddd25b6-6227-470a-8d3d-c1748e15c2f1\") " pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:48.811331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.811288 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7320-6a2244" podUID="aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" containerName="test-curl-7320-6a2244" containerID="cri-o://0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f" gracePeriod=2 Oct 30 22:16:48.842177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.842154 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psffw\" (UniqueName: \"kubernetes.io/projected/2ddd25b6-6227-470a-8d3d-c1748e15c2f1-kube-api-access-psffw\") pod \"test-curl-7320-18cefc\" (UID: \"2ddd25b6-6227-470a-8d3d-c1748e15c2f1\") " pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:48.863514 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:48.863460 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-6a2244\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:48.894111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:48.894088 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:49.286535 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:49.281535 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ddd25b6_6227_470a_8d3d_c1748e15c2f1.slice/crio-ffabd3d702b945cd4108110296d329bd621a8e11a36488ba207d69fc59aa8d28 WatchSource:0}: Error finding container ffabd3d702b945cd4108110296d329bd621a8e11a36488ba207d69fc59aa8d28: Status 404 returned error can't find the container with id ffabd3d702b945cd4108110296d329bd621a8e11a36488ba207d69fc59aa8d28 Oct 30 22:16:49.286535 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:49.283777 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-18cefc"] Oct 30 22:16:49.819442 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:49.819411 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-18cefc" event={"ID":"2ddd25b6-6227-470a-8d3d-c1748e15c2f1","Type":"ContainerStarted","Data":"9d3d0a1518c81717eea7fdbabe38cd23b08e1fe3fb1add45b18afd800b18cef6"} Oct 30 22:16:49.819949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:49.819451 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-18cefc" event={"ID":"2ddd25b6-6227-470a-8d3d-c1748e15c2f1","Type":"ContainerStarted","Data":"ffabd3d702b945cd4108110296d329bd621a8e11a36488ba207d69fc59aa8d28"} Oct 30 22:16:49.936997 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:49.936931 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7320-18cefc" podStartSLOduration=1.936913334 podStartE2EDuration="1.936913334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:49.933738076 +0000 UTC m=+2058.614887973" watchObservedRunningTime="2026-10-30 22:16:49.936913334 +0000 UTC m=+2058.618063229" Oct 30 22:16:50.839621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:50.839292 4933 generic.go:356] "Generic (PLEG): container finished" podID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerID="962fb143d1a66dea0b8186d64a586ed7c131ac6d6dc264af48bbb0b52894ce12" exitCode=0 Oct 30 22:16:50.839621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:50.839473 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" event={"ID":"adcaa605-fa45-4f88-ac6f-12a4d9b4363b","Type":"ContainerDied","Data":"962fb143d1a66dea0b8186d64a586ed7c131ac6d6dc264af48bbb0b52894ce12"} Oct 30 22:16:50.843168 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:50.842650 4933 generic.go:356] "Generic (PLEG): container finished" podID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerID="0f1c6ba2ec315b6ba3c265490bf523b2abf26065c6409f87e16b0b7854e12484" exitCode=0 Oct 30 22:16:50.843888 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:50.843741 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" event={"ID":"26975667-9851-4f05-a4ff-a9ea7e08c769","Type":"ContainerDied","Data":"0f1c6ba2ec315b6ba3c265490bf523b2abf26065c6409f87e16b0b7854e12484"} Oct 30 22:16:51.029566 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.029535 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:16:51.096042 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.096015 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:16:51.123734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.123701 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-podinfo\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.123815 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.123787 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-token\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-token\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.123884 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.123869 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-credential-socket\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.123941 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.123923 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-certs\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.124001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.123991 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-kube-api-access-fh59r\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-kube-api-access-fh59r\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.124057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124036 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-data\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.124146 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124129 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-envoy\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.124221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124123 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-credential-socket" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.124221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124191 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-certs" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.124331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124257 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-socket\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.124331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124321 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26975667-9851-4f05-a4ff-a9ea7e08c769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26975667-9851-4f05-a4ff-a9ea7e08c769-istiod-ca-cert\") pod \"26975667-9851-4f05-a4ff-a9ea7e08c769\" (UID: \"26975667-9851-4f05-a4ff-a9ea7e08c769\") " Oct 30 22:16:51.124539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124386 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-data" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.124704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124583 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-socket" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.125012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124930 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.125012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124944 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26975667-9851-4f05-a4ff-a9ea7e08c769-istiod-ca-cert" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:51.125012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124956 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.125012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124974 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.125012 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.124992 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.126182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.126153 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-token" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:51.126283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.126264 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-kube-api-access-fh59r" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "kube-api-access-fh59r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:51.126416 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.126396 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-podinfo" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:51.126796 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.126778 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-envoy" pod "26975667-9851-4f05-a4ff-a9ea7e08c769" (UID: "26975667-9851-4f05-a4ff-a9ea7e08c769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.139964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.139939 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:51.225816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.225788 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-credential-socket\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.225985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.225856 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-certs\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.225985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.225876 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-podinfo\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.225985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.225949 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-data\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.226154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226015 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-socket\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.226154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226108 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istiod-ca-cert\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.226265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226090 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-credential-socket" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.226265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226146 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-certs" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.226265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226156 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-kube-api-access-kgnc5\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-kube-api-access-kgnc5\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.226424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226279 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-socket" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.226424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226288 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-data" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.226424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226289 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-token\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-token\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.226424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226331 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-envoy\") pod \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\" (UID: \"adcaa605-fa45-4f88-ac6f-12a4d9b4363b\") " Oct 30 22:16:51.226424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226404 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7-kube-api-access-dqnmc\" (UniqueName: \"kubernetes.io/projected/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7-kube-api-access-dqnmc\") pod \"aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7\" (UID: \"aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7\") " Oct 30 22:16:51.226712 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.226420 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istiod-ca-cert" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:51.227184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227159 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227190 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh59r\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-kube-api-access-fh59r\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227207 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227223 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227239 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227255 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26975667-9851-4f05-a4ff-a9ea7e08c769-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227269 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227283 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227300 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227298 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26975667-9851-4f05-a4ff-a9ea7e08c769-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.227786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.227312 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.228829 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.228803 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-podinfo" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:51.229074 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.229043 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-token" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:51.229157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.229069 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-kube-api-access-kgnc5" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "kube-api-access-kgnc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:51.229157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.229136 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-envoy" pod "adcaa605-fa45-4f88-ac6f-12a4d9b4363b" (UID: "adcaa605-fa45-4f88-ac6f-12a4d9b4363b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:51.229774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.229753 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7-kube-api-access-dqnmc" pod "aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" (UID: "aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7"). InnerVolumeSpecName "kube-api-access-dqnmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:51.328922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.328889 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.328922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.328921 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgnc5\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-kube-api-access-kgnc5\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.329117 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.328935 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.329117 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.328947 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adcaa605-fa45-4f88-ac6f-12a4d9b4363b-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.329117 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.328960 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqnmc\" (UniqueName: \"kubernetes.io/projected/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7-kube-api-access-dqnmc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:51.418873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.418804 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-18cefc"] Oct 30 22:16:51.422698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.422671 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-18cefc"] Oct 30 22:16:51.509631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.507811 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" path="/var/lib/kubelet/pods/aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7/volumes" Oct 30 22:16:51.578502 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.578469 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-9a1b82"] Oct 30 22:16:51.579038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579002 4933 cpu_manager.go:414] "containerMap: removing container" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" Oct 30 22:16:51.579038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579018 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" Oct 30 22:16:51.579122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579045 4933 cpu_manager.go:414] "containerMap: removing container" podUID="2ddd25b6-6227-470a-8d3d-c1748e15c2f1" containerName="test-curl-7320-18cefc" Oct 30 22:16:51.579122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579050 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddd25b6-6227-470a-8d3d-c1748e15c2f1" containerName="test-curl-7320-18cefc" Oct 30 22:16:51.579122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579060 4933 cpu_manager.go:414] "containerMap: removing container" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" Oct 30 22:16:51.579122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579064 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" Oct 30 22:16:51.579244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579158 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddd25b6-6227-470a-8d3d-c1748e15c2f1" containerName="test-curl-7320-18cefc" Oct 30 22:16:51.579244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579168 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" containerName="istio-proxy" Oct 30 22:16:51.579244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.579179 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" containerName="istio-proxy" Oct 30 22:16:51.583733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.583718 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:51.637368 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.637339 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-9a1b82"] Oct 30 22:16:51.736106 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.736072 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58sxn\" (UniqueName: \"kubernetes.io/projected/314f986e-bba8-45c8-9374-8bf7eeee523f-kube-api-access-58sxn\") pod \"test-curl-7320-9a1b82\" (UID: \"314f986e-bba8-45c8-9374-8bf7eeee523f\") " pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:51.837150 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.837070 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58sxn\" (UniqueName: \"kubernetes.io/projected/314f986e-bba8-45c8-9374-8bf7eeee523f-kube-api-access-58sxn\") pod \"test-curl-7320-9a1b82\" (UID: \"314f986e-bba8-45c8-9374-8bf7eeee523f\") " pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:51.852904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.852871 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" Oct 30 22:16:51.852904 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.852899 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x" event={"ID":"adcaa605-fa45-4f88-ac6f-12a4d9b4363b","Type":"ContainerDied","Data":"0f09906fcdd1ef8d2b620d50c6f6ae105e8f3ff9a4ba34be9dcb81b4f5bb0bc7"} Oct 30 22:16:51.853454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.852963 4933 scope.go:122] "RemoveContainer" containerID="962fb143d1a66dea0b8186d64a586ed7c131ac6d6dc264af48bbb0b52894ce12" Oct 30 22:16:51.855618 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.855310 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" Oct 30 22:16:51.855618 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.855340 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb" event={"ID":"26975667-9851-4f05-a4ff-a9ea7e08c769","Type":"ContainerDied","Data":"eea01da825f7f71cd14152a38a60fb6d411c67cb1915c797f171cd6289345fd4"} Oct 30 22:16:51.857031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.857007 4933 generic.go:356] "Generic (PLEG): container finished" podID="aeeb5b3c-cb65-4f56-9557-9e4179d2d4c7" containerID="0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f" exitCode=137 Oct 30 22:16:51.857134 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.857102 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-6a2244" Oct 30 22:16:51.857366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.857313 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7320-18cefc" podUID="2ddd25b6-6227-470a-8d3d-c1748e15c2f1" containerName="test-curl-7320-18cefc" containerID="cri-o://9d3d0a1518c81717eea7fdbabe38cd23b08e1fe3fb1add45b18afd800b18cef6" gracePeriod=2 Oct 30 22:16:51.865980 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.865947 4933 scope.go:122] "RemoveContainer" containerID="0f1c6ba2ec315b6ba3c265490bf523b2abf26065c6409f87e16b0b7854e12484" Oct 30 22:16:51.875388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.875368 4933 scope.go:122] "RemoveContainer" containerID="0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f" Oct 30 22:16:51.884298 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.884280 4933 scope.go:122] "RemoveContainer" containerID="0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f" Oct 30 22:16:51.884633 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:51.884613 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f\": container with ID starting with 0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f not found: ID does not exist" containerID="0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f" Oct 30 22:16:51.884761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.884640 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f"} err="failed to get container status \"0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f\": rpc error: code = NotFound desc = could not find container \"0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f\": container with ID starting with 0f7fc51db4e38aeaa9a6a1e1472847ef9c85af9387584ac17cb76af30618ce7f not found: ID does not exist" Oct 30 22:16:51.895219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.895194 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58sxn\" (UniqueName: \"kubernetes.io/projected/314f986e-bba8-45c8-9374-8bf7eeee523f-kube-api-access-58sxn\") pod \"test-curl-7320-9a1b82\" (UID: \"314f986e-bba8-45c8-9374-8bf7eeee523f\") " pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:51.896115 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:51.896095 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:52.027208 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.027175 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:16:52.066922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.066792 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x"] Oct 30 22:16:52.129536 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.123700 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-5xm5x"] Oct 30 22:16:52.210399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.210290 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb"] Oct 30 22:16:52.214272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.214244 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-9t7qb"] Oct 30 22:16:52.311185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.311158 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-9a1b82"] Oct 30 22:16:52.312295 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:52.312263 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod314f986e_bba8_45c8_9374_8bf7eeee523f.slice/crio-99cf3fc3b145b1b9c9aa861b1df45a63ad1108dc06a40ee2a4cdb36032974eb3 WatchSource:0}: Error finding container 99cf3fc3b145b1b9c9aa861b1df45a63ad1108dc06a40ee2a4cdb36032974eb3: Status 404 returned error can't find the container with id 99cf3fc3b145b1b9c9aa861b1df45a63ad1108dc06a40ee2a4cdb36032974eb3 Oct 30 22:16:52.998971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.998726 4933 generic.go:356] "Generic (PLEG): container finished" podID="4b949074-23f2-4c8b-bd61-d5231364de34" containerID="d4f5ec9a84413291bb4bccd02755d49f980b41780fac98ad6495f0878373fb05" exitCode=0 Oct 30 22:16:52.998971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.998842 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" event={"ID":"4b949074-23f2-4c8b-bd61-d5231364de34","Type":"ContainerDied","Data":"d4f5ec9a84413291bb4bccd02755d49f980b41780fac98ad6495f0878373fb05"} Oct 30 22:16:52.998971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.998892 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" event={"ID":"4b949074-23f2-4c8b-bd61-d5231364de34","Type":"ContainerDied","Data":"8e5682638c238eb100ff59ca071873fa8ec02f44a3eb01abb943b889ab9c2151"} Oct 30 22:16:52.998971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.998917 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e5682638c238eb100ff59ca071873fa8ec02f44a3eb01abb943b889ab9c2151" Oct 30 22:16:52.999649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:52.999637 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:16:53.002556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.002226 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-9a1b82" event={"ID":"314f986e-bba8-45c8-9374-8bf7eeee523f","Type":"ContainerStarted","Data":"1572d094c214cb79797d40368a90bacf32b0ed5a4389c15e36168ad55c41fb5a"} Oct 30 22:16:53.002556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.002266 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-9a1b82" event={"ID":"314f986e-bba8-45c8-9374-8bf7eeee523f","Type":"ContainerStarted","Data":"99cf3fc3b145b1b9c9aa861b1df45a63ad1108dc06a40ee2a4cdb36032974eb3"} Oct 30 22:16:53.015511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015063 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-credential-socket\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015163 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-socket\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015244 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b949074-23f2-4c8b-bd61-d5231364de34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b949074-23f2-4c8b-bd61-d5231364de34-istio-podinfo\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015302 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-certs\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015334 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-data\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015388 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b949074-23f2-4c8b-bd61-d5231364de34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b949074-23f2-4c8b-bd61-d5231364de34-istiod-ca-cert\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015419 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-envoy\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015535 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-kube-api-access-xkpkk\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-kube-api-access-xkpkk\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.015957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.015576 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-istio-token\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-istio-token\") pod \"4b949074-23f2-4c8b-bd61-d5231364de34\" (UID: \"4b949074-23f2-4c8b-bd61-d5231364de34\") " Oct 30 22:16:53.021539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.018985 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-certs" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:53.021539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.019315 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-credential-socket" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:53.021539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.019578 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-socket" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:53.021539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.020038 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-data" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:53.021539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.020416 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b949074-23f2-4c8b-bd61-d5231364de34-istiod-ca-cert" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:16:53.044832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.044791 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-istio-token" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:53.045623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.045581 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-kube-api-access-xkpkk" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "kube-api-access-xkpkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:53.072053 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.071676 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b949074-23f2-4c8b-bd61-d5231364de34-istio-podinfo" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:16:53.072638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.072606 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-envoy" pod "4b949074-23f2-4c8b-bd61-d5231364de34" (UID: "4b949074-23f2-4c8b-bd61-d5231364de34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.117927 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.117961 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b949074-23f2-4c8b-bd61-d5231364de34-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.117974 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.117986 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.118000 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b949074-23f2-4c8b-bd61-d5231364de34-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.118013 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.118024 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpkk\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-kube-api-access-xkpkk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.118038 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b949074-23f2-4c8b-bd61-d5231364de34-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.118511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.118053 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b949074-23f2-4c8b-bd61-d5231364de34-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:53.264121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.264054 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7320-9a1b82" podStartSLOduration=2.2640365669999998 podStartE2EDuration="2.264036567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:53.263141125 +0000 UTC m=+2061.944291018" watchObservedRunningTime="2026-10-30 22:16:53.264036567 +0000 UTC m=+2061.945186461" Oct 30 22:16:53.468924 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.468849 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26975667-9851-4f05-a4ff-a9ea7e08c769" path="/var/lib/kubelet/pods/26975667-9851-4f05-a4ff-a9ea7e08c769/volumes" Oct 30 22:16:53.469322 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:53.469309 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adcaa605-fa45-4f88-ac6f-12a4d9b4363b" path="/var/lib/kubelet/pods/adcaa605-fa45-4f88-ac6f-12a4d9b4363b/volumes" Oct 30 22:16:54.018811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.018768 4933 generic.go:356] "Generic (PLEG): container finished" podID="2ddd25b6-6227-470a-8d3d-c1748e15c2f1" containerID="9d3d0a1518c81717eea7fdbabe38cd23b08e1fe3fb1add45b18afd800b18cef6" exitCode=137 Oct 30 22:16:54.019273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.019253 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8" Oct 30 22:16:54.092170 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.092139 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:54.131602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.131572 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddd25b6-6227-470a-8d3d-c1748e15c2f1-kube-api-access-psffw\" (UniqueName: \"kubernetes.io/projected/2ddd25b6-6227-470a-8d3d-c1748e15c2f1-kube-api-access-psffw\") pod \"2ddd25b6-6227-470a-8d3d-c1748e15c2f1\" (UID: \"2ddd25b6-6227-470a-8d3d-c1748e15c2f1\") " Oct 30 22:16:54.133963 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.133927 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddd25b6-6227-470a-8d3d-c1748e15c2f1-kube-api-access-psffw" pod "2ddd25b6-6227-470a-8d3d-c1748e15c2f1" (UID: "2ddd25b6-6227-470a-8d3d-c1748e15c2f1"). InnerVolumeSpecName "kube-api-access-psffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:54.233237 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.233194 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psffw\" (UniqueName: \"kubernetes.io/projected/2ddd25b6-6227-470a-8d3d-c1748e15c2f1-kube-api-access-psffw\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:54.241822 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.241748 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8"] Oct 30 22:16:54.292182 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.292151 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-4csn8"] Oct 30 22:16:54.629647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.629620 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-9a1b82"] Oct 30 22:16:54.634284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.634258 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-9a1b82"] Oct 30 22:16:54.795515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.795472 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7320-7d9ef8"] Oct 30 22:16:54.796283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.796258 4933 cpu_manager.go:414] "containerMap: removing container" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" Oct 30 22:16:54.796283 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.796282 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" Oct 30 22:16:54.796534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.796365 4933 cpu_manager.go:414] "containerMap: removing container" podUID="314f986e-bba8-45c8-9374-8bf7eeee523f" containerName="test-curl-7320-9a1b82" Oct 30 22:16:54.796534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.796374 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314f986e-bba8-45c8-9374-8bf7eeee523f" containerName="test-curl-7320-9a1b82" Oct 30 22:16:54.796654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.796540 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="314f986e-bba8-45c8-9374-8bf7eeee523f" containerName="test-curl-7320-9a1b82" Oct 30 22:16:54.796654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.796558 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" containerName="istio-proxy" Oct 30 22:16:54.800120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.800097 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:54.840512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.840455 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s69g\" (UniqueName: \"kubernetes.io/projected/2feaea2c-4e5f-42fa-9f9b-96e739a775ad-kube-api-access-4s69g\") pod \"test-curl-7320-7d9ef8\" (UID: \"2feaea2c-4e5f-42fa-9f9b-96e739a775ad\") " pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:54.877881 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.873460 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-7d9ef8"] Oct 30 22:16:54.942382 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:54.942287 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s69g\" (UniqueName: \"kubernetes.io/projected/2feaea2c-4e5f-42fa-9f9b-96e739a775ad-kube-api-access-4s69g\") pod \"test-curl-7320-7d9ef8\" (UID: \"2feaea2c-4e5f-42fa-9f9b-96e739a775ad\") " pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:55.000698 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.000672 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s69g\" (UniqueName: \"kubernetes.io/projected/2feaea2c-4e5f-42fa-9f9b-96e739a775ad-kube-api-access-4s69g\") pod \"test-curl-7320-7d9ef8\" (UID: \"2feaea2c-4e5f-42fa-9f9b-96e739a775ad\") " pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:55.029642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.029609 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:55.029642 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.029638 4933 scope.go:122] "RemoveContainer" containerID="9d3d0a1518c81717eea7fdbabe38cd23b08e1fe3fb1add45b18afd800b18cef6" Oct 30 22:16:55.030210 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.029861 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7320-9a1b82" podUID="314f986e-bba8-45c8-9374-8bf7eeee523f" containerName="test-curl-7320-9a1b82" containerID="cri-o://1572d094c214cb79797d40368a90bacf32b0ed5a4389c15e36168ad55c41fb5a" gracePeriod=2 Oct 30 22:16:55.091063 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:55.090736 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-9a1b82\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="314f986e-bba8-45c8-9374-8bf7eeee523f" pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:55.119689 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.119375 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:55.146966 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:55.142715 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-9a1b82\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="314f986e-bba8-45c8-9374-8bf7eeee523f" pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:55.194791 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:55.194717 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-18cefc\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="2ddd25b6-6227-470a-8d3d-c1748e15c2f1" pod="opendatahub/test-curl-7320-18cefc" Oct 30 22:16:55.465783 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.465710 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddd25b6-6227-470a-8d3d-c1748e15c2f1" path="/var/lib/kubelet/pods/2ddd25b6-6227-470a-8d3d-c1748e15c2f1/volumes" Oct 30 22:16:55.466215 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.466200 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b949074-23f2-4c8b-bd61-d5231364de34" path="/var/lib/kubelet/pods/4b949074-23f2-4c8b-bd61-d5231364de34/volumes" Oct 30 22:16:55.535880 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:55.535850 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2feaea2c_4e5f_42fa_9f9b_96e739a775ad.slice/crio-05ae6b24743bb4de8c2733ce685a227929e3709bac111d24fa1ae662a275dfa3 WatchSource:0}: Error finding container 05ae6b24743bb4de8c2733ce685a227929e3709bac111d24fa1ae662a275dfa3: Status 404 returned error can't find the container with id 05ae6b24743bb4de8c2733ce685a227929e3709bac111d24fa1ae662a275dfa3 Oct 30 22:16:55.536120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:55.536100 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7320-7d9ef8"] Oct 30 22:16:56.039585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:56.039536 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-7d9ef8" event={"ID":"2feaea2c-4e5f-42fa-9f9b-96e739a775ad","Type":"ContainerStarted","Data":"e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7"} Oct 30 22:16:56.039585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:56.039574 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7320-7d9ef8" event={"ID":"2feaea2c-4e5f-42fa-9f9b-96e739a775ad","Type":"ContainerStarted","Data":"05ae6b24743bb4de8c2733ce685a227929e3709bac111d24fa1ae662a275dfa3"} Oct 30 22:16:56.154132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:56.154082 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7320-7d9ef8" podStartSLOduration=2.154068895 podStartE2EDuration="2.154068895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:56.152369007 +0000 UTC m=+2064.833518900" watchObservedRunningTime="2026-10-30 22:16:56.154068895 +0000 UTC m=+2064.835218787" Oct 30 22:16:56.832010 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:56.831979 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf"] Oct 30 22:16:56.882579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:56.882552 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d516c4-nz5bf"] Oct 30 22:16:57.050020 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.049995 4933 generic.go:356] "Generic (PLEG): container finished" podID="314f986e-bba8-45c8-9374-8bf7eeee523f" containerID="1572d094c214cb79797d40368a90bacf32b0ed5a4389c15e36168ad55c41fb5a" exitCode=137 Oct 30 22:16:57.244621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.244599 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:57.281801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.281776 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314f986e-bba8-45c8-9374-8bf7eeee523f-kube-api-access-58sxn\" (UniqueName: \"kubernetes.io/projected/314f986e-bba8-45c8-9374-8bf7eeee523f-kube-api-access-58sxn\") pod \"314f986e-bba8-45c8-9374-8bf7eeee523f\" (UID: \"314f986e-bba8-45c8-9374-8bf7eeee523f\") " Oct 30 22:16:57.283915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.283889 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314f986e-bba8-45c8-9374-8bf7eeee523f-kube-api-access-58sxn" pod "314f986e-bba8-45c8-9374-8bf7eeee523f" (UID: "314f986e-bba8-45c8-9374-8bf7eeee523f"). InnerVolumeSpecName "kube-api-access-58sxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:16:57.383248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.383222 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58sxn\" (UniqueName: \"kubernetes.io/projected/314f986e-bba8-45c8-9374-8bf7eeee523f-kube-api-access-58sxn\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:16:57.466200 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.466140 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314f986e-bba8-45c8-9374-8bf7eeee523f" path="/var/lib/kubelet/pods/314f986e-bba8-45c8-9374-8bf7eeee523f/volumes" Oct 30 22:16:57.476167 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.467183 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf216553-f6e3-407d-9f44-2e2fd786f836" path="/var/lib/kubelet/pods/bf216553-f6e3-407d-9f44-2e2fd786f836/volumes" Oct 30 22:16:57.616230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.616189 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7320-7d9ef8"] Oct 30 22:16:57.621713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:57.621688 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7320-7d9ef8"] Oct 30 22:16:58.063231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.063203 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-9a1b82" Oct 30 22:16:58.063231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.063226 4933 scope.go:122] "RemoveContainer" containerID="1572d094c214cb79797d40368a90bacf32b0ed5a4389c15e36168ad55c41fb5a" Oct 30 22:16:58.063739 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.063424 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7320-7d9ef8" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" containerName="test-curl-7320-7d9ef8" containerID="cri-o://e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7" gracePeriod=2 Oct 30 22:16:58.115189 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:58.115162 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-7d9ef8\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:58.166842 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:58.166812 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-7d9ef8\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:58.189250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.189223 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg"] Oct 30 22:16:58.189742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.189730 4933 cpu_manager.go:414] "containerMap: removing container" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" containerName="test-curl-7320-7d9ef8" Oct 30 22:16:58.189797 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.189744 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" containerName="test-curl-7320-7d9ef8" Oct 30 22:16:58.189863 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.189854 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" containerName="test-curl-7320-7d9ef8" Oct 30 22:16:58.194718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.194532 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.246626 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:58.246593 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-7d9ef8\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:16:58.247879 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.247847 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:16:58.250293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.250271 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg"] Oct 30 22:16:58.293185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.293162 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ntb6\" (UniqueName: \"kubernetes.io/projected/8c797faa-b135-4781-a37d-0861c755a8eb-kube-api-access-6ntb6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.293358 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.293339 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.293475 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.293459 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.296417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.296401 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:16:58.296465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.296443 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:16:58.296520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.296469 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:16:58.395098 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.395033 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.395098 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.395076 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ntb6\" (UniqueName: \"kubernetes.io/projected/8c797faa-b135-4781-a37d-0861c755a8eb-kube-api-access-6ntb6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.395265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.395165 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.395674 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.395657 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.397681 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.397661 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.453049 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.453027 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ntb6\" (UniqueName: \"kubernetes.io/projected/8c797faa-b135-4781-a37d-0861c755a8eb-kube-api-access-6ntb6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.507930 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.507908 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:16:58.550908 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:16:58.550878 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:16:58.893623 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:16:58.893593 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c797faa_b135_4781_a37d_0861c755a8eb.slice/crio-e967cd09fa335dabf411a94832943feef3012757ff7a782c40b33b214b1f2065 WatchSource:0}: Error finding container e967cd09fa335dabf411a94832943feef3012757ff7a782c40b33b214b1f2065: Status 404 returned error can't find the container with id e967cd09fa335dabf411a94832943feef3012757ff7a782c40b33b214b1f2065 Oct 30 22:16:58.895240 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:58.895216 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg"] Oct 30 22:16:59.071643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:16:59.071614 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" event={"ID":"8c797faa-b135-4781-a37d-0861c755a8eb","Type":"ContainerStarted","Data":"e967cd09fa335dabf411a94832943feef3012757ff7a782c40b33b214b1f2065"} Oct 30 22:17:00.081873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:00.081845 4933 generic.go:356] "Generic (PLEG): container finished" podID="8c797faa-b135-4781-a37d-0861c755a8eb" containerID="b1d045ef1457cc73f4363950cf6fb2d9cf880f892f472f905c35c9c089a5bff6" exitCode=0 Oct 30 22:17:00.082241 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:00.081937 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" event={"ID":"8c797faa-b135-4781-a37d-0861c755a8eb","Type":"ContainerDied","Data":"b1d045ef1457cc73f4363950cf6fb2d9cf880f892f472f905c35c9c089a5bff6"} Oct 30 22:17:00.305902 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:00.305880 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:17:00.417955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:00.417891 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2feaea2c-4e5f-42fa-9f9b-96e739a775ad-kube-api-access-4s69g\" (UniqueName: \"kubernetes.io/projected/2feaea2c-4e5f-42fa-9f9b-96e739a775ad-kube-api-access-4s69g\") pod \"2feaea2c-4e5f-42fa-9f9b-96e739a775ad\" (UID: \"2feaea2c-4e5f-42fa-9f9b-96e739a775ad\") " Oct 30 22:17:00.420120 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:00.420094 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2feaea2c-4e5f-42fa-9f9b-96e739a775ad-kube-api-access-4s69g" pod "2feaea2c-4e5f-42fa-9f9b-96e739a775ad" (UID: "2feaea2c-4e5f-42fa-9f9b-96e739a775ad"). InnerVolumeSpecName "kube-api-access-4s69g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:00.519921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:00.519887 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s69g\" (UniqueName: \"kubernetes.io/projected/2feaea2c-4e5f-42fa-9f9b-96e739a775ad-kube-api-access-4s69g\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:01.093575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.093543 4933 generic.go:356] "Generic (PLEG): container finished" podID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" containerID="e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7" exitCode=137 Oct 30 22:17:01.094031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.093603 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:17:01.094031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.093651 4933 scope.go:122] "RemoveContainer" containerID="e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7" Oct 30 22:17:01.110916 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.110898 4933 scope.go:122] "RemoveContainer" containerID="e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7" Oct 30 22:17:01.111207 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:17:01.111185 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7\": container with ID starting with e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7 not found: ID does not exist" containerID="e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7" Oct 30 22:17:01.111275 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.111218 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7"} err="failed to get container status \"e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7\": rpc error: code = NotFound desc = could not find container \"e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7\": container with ID starting with e9f6a06d11ad0e048fa3a6d6f1f076dd0e4e4a163e0c0d2615d68be44912caf7 not found: ID does not exist" Oct 30 22:17:01.161716 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:17:01.160125 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-7d9ef8\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:17:01.316966 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.316937 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:17:01.435643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.435548 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-xks-ca\") pod \"8c797faa-b135-4781-a37d-0861c755a8eb\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " Oct 30 22:17:01.435823 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.435648 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c797faa-b135-4781-a37d-0861c755a8eb-kube-api-access-6ntb6\" (UniqueName: \"kubernetes.io/projected/8c797faa-b135-4781-a37d-0861c755a8eb-kube-api-access-6ntb6\") pod \"8c797faa-b135-4781-a37d-0861c755a8eb\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " Oct 30 22:17:01.435823 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.435705 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-openshift-ca\") pod \"8c797faa-b135-4781-a37d-0861c755a8eb\" (UID: \"8c797faa-b135-4781-a37d-0861c755a8eb\") " Oct 30 22:17:01.436543 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.436481 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-openshift-ca" pod "8c797faa-b135-4781-a37d-0861c755a8eb" (UID: "8c797faa-b135-4781-a37d-0861c755a8eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:01.439195 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.438912 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c797faa-b135-4781-a37d-0861c755a8eb-kube-api-access-6ntb6" pod "8c797faa-b135-4781-a37d-0861c755a8eb" (UID: "8c797faa-b135-4781-a37d-0861c755a8eb"). InnerVolumeSpecName "kube-api-access-6ntb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:01.441982 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.441947 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-xks-ca" pod "8c797faa-b135-4781-a37d-0861c755a8eb" (UID: "8c797faa-b135-4781-a37d-0861c755a8eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:01.471736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.471559 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" path="/var/lib/kubelet/pods/2feaea2c-4e5f-42fa-9f9b-96e739a775ad/volumes" Oct 30 22:17:01.491511 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:17:01.490819 4933 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7320-7d9ef8\" is forbidden: User \"system:node:ip-10-0-131-250.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-250.us-west-1.compute.internal' and this object" podUID="2feaea2c-4e5f-42fa-9f9b-96e739a775ad" pod="opendatahub/test-curl-7320-7d9ef8" Oct 30 22:17:01.537870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.537805 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:01.537870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.537836 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ntb6\" (UniqueName: \"kubernetes.io/projected/8c797faa-b135-4781-a37d-0861c755a8eb-kube-api-access-6ntb6\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:01.537870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:01.537848 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c797faa-b135-4781-a37d-0861c755a8eb-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:02.106017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:02.105982 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" event={"ID":"8c797faa-b135-4781-a37d-0861c755a8eb","Type":"ContainerDied","Data":"e967cd09fa335dabf411a94832943feef3012757ff7a782c40b33b214b1f2065"} Oct 30 22:17:02.106456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:02.106039 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e967cd09fa335dabf411a94832943feef3012757ff7a782c40b33b214b1f2065" Oct 30 22:17:02.106456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:02.106001 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg" Oct 30 22:17:03.303077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:03.303045 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk"] Oct 30 22:17:03.355736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:03.355706 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-qtpzk"] Oct 30 22:17:03.467705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:03.467677 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fda1b43-5818-458d-b856-49e01cf96286" path="/var/lib/kubelet/pods/0fda1b43-5818-458d-b856-49e01cf96286/volumes" Oct 30 22:17:03.867560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:03.866267 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p"] Oct 30 22:17:03.867560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:03.866568 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" containerID="cri-o://c4c9d425d65eb676a55e977d1e1de52993d9449a59e5f9506108f7b96a6e72ef" gracePeriod=30 Oct 30 22:17:07.845750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.845715 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2"] Oct 30 22:17:07.846267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.846251 4933 cpu_manager.go:414] "containerMap: removing container" podUID="8c797faa-b135-4781-a37d-0861c755a8eb" containerName="revoke-keys" Oct 30 22:17:07.846267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.846268 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c797faa-b135-4781-a37d-0861c755a8eb" containerName="revoke-keys" Oct 30 22:17:07.846387 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.846375 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c797faa-b135-4781-a37d-0861c755a8eb" containerName="revoke-keys" Oct 30 22:17:07.851592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.851572 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp"] Oct 30 22:17:07.851729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.851708 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:07.855456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.855438 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:07.905117 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.905095 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:07.905341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.905325 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:07.922239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.922213 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:07.922362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.922255 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55hm\" (UniqueName: \"kubernetes.io/projected/7f58c787-5475-4f57-ae80-2ece9ae38bda-kube-api-access-f55hm\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:07.922362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.922331 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:07.922454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.922412 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:07.922454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.922442 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:07.922539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.922526 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rcm4\" (UniqueName: \"kubernetes.io/projected/0c75fcfd-fac3-45d6-a99c-9519c793396e-kube-api-access-7rcm4\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:07.967507 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.961312 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:17:07.967507 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.961621 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:07.974534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.974478 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2"] Oct 30 22:17:07.979423 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:07.979396 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp"] Oct 30 22:17:08.024308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.024274 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:08.024465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.024414 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcm4\" (UniqueName: \"kubernetes.io/projected/0c75fcfd-fac3-45d6-a99c-9519c793396e-kube-api-access-7rcm4\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:08.024539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.024522 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:08.024596 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.024569 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55hm\" (UniqueName: \"kubernetes.io/projected/7f58c787-5475-4f57-ae80-2ece9ae38bda-kube-api-access-f55hm\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:08.024652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.024608 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:08.024759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.024738 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:08.025202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.025175 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:08.025453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.025429 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:08.027003 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.026984 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:08.027215 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.027193 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:08.083701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.083670 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcm4\" (UniqueName: \"kubernetes.io/projected/0c75fcfd-fac3-45d6-a99c-9519c793396e-kube-api-access-7rcm4\") pod \"maas-api-revoke-sub-539a2043-xjchp\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:08.085106 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.085077 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55hm\" (UniqueName: \"kubernetes.io/projected/7f58c787-5475-4f57-ae80-2ece9ae38bda-kube-api-access-f55hm\") pod \"maas-api-revoke-sub-0abedc8a-vfxm2\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:08.171793 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.171698 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:08.181425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.181401 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:08.615676 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.614177 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2"] Oct 30 22:17:08.617461 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:17:08.617408 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f58c787_5475_4f57_ae80_2ece9ae38bda.slice/crio-e67f7d5255a9170b1baa4c430cb19f718b7dfa23b129096679e77a45e57c1c84 WatchSource:0}: Error finding container e67f7d5255a9170b1baa4c430cb19f718b7dfa23b129096679e77a45e57c1c84: Status 404 returned error can't find the container with id e67f7d5255a9170b1baa4c430cb19f718b7dfa23b129096679e77a45e57c1c84 Oct 30 22:17:08.667542 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:17:08.667503 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c75fcfd_fac3_45d6_a99c_9519c793396e.slice/crio-58d901b3568565fb45d6c8e09d64723448aef1f6740980da74c51a58186e12f6 WatchSource:0}: Error finding container 58d901b3568565fb45d6c8e09d64723448aef1f6740980da74c51a58186e12f6: Status 404 returned error can't find the container with id 58d901b3568565fb45d6c8e09d64723448aef1f6740980da74c51a58186e12f6 Oct 30 22:17:08.669677 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:08.669655 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp"] Oct 30 22:17:09.298511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.294891 4933 generic.go:356] "Generic (PLEG): container finished" podID="f015183f-507c-4318-af33-2190e83755e8" containerID="c4c9d425d65eb676a55e977d1e1de52993d9449a59e5f9506108f7b96a6e72ef" exitCode=0 Oct 30 22:17:09.298511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.295015 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" event={"ID":"f015183f-507c-4318-af33-2190e83755e8","Type":"ContainerDied","Data":"c4c9d425d65eb676a55e977d1e1de52993d9449a59e5f9506108f7b96a6e72ef"} Oct 30 22:17:09.298511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.296655 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" event={"ID":"0c75fcfd-fac3-45d6-a99c-9519c793396e","Type":"ContainerStarted","Data":"2e73c3eb0fc11a2a34ee4f332074416841b63631076ed05e6154ecbbe9fb49c7"} Oct 30 22:17:09.298511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.296689 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" event={"ID":"0c75fcfd-fac3-45d6-a99c-9519c793396e","Type":"ContainerStarted","Data":"58d901b3568565fb45d6c8e09d64723448aef1f6740980da74c51a58186e12f6"} Oct 30 22:17:09.299171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.299092 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" event={"ID":"7f58c787-5475-4f57-ae80-2ece9ae38bda","Type":"ContainerStarted","Data":"d4defa57fa7e8ff7815f69c03aecc24e4a0ffdb17e342bc925b22212cf92f1fd"} Oct 30 22:17:09.299171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.299126 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" event={"ID":"7f58c787-5475-4f57-ae80-2ece9ae38bda","Type":"ContainerStarted","Data":"e67f7d5255a9170b1baa4c430cb19f718b7dfa23b129096679e77a45e57c1c84"} Oct 30 22:17:09.366866 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.366840 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 22:17:09.418796 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.418737 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" podStartSLOduration=2.418716835 podStartE2EDuration="2.418716835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:09.414996248 +0000 UTC m=+2078.096146142" watchObservedRunningTime="2026-10-30 22:17:09.418716835 +0000 UTC m=+2078.099866729" Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463434 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-socket\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463541 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f015183f-507c-4318-af33-2190e83755e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f015183f-507c-4318-af33-2190e83755e8-istiod-ca-cert\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463572 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-data\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463628 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-certs\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463667 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-envoy\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463714 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-kube-api-access-dhv7q\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-kube-api-access-dhv7q\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463781 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f015183f-507c-4318-af33-2190e83755e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f015183f-507c-4318-af33-2190e83755e8-istio-podinfo\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463816 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-credential-socket\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.463928 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-istio-token\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-istio-token\") pod \"f015183f-507c-4318-af33-2190e83755e8\" (UID: \"f015183f-507c-4318-af33-2190e83755e8\") " Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.467469 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-socket" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.467830 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-data" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.478518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.468139 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-credential-socket" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.488532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.482968 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f015183f-507c-4318-af33-2190e83755e8-istiod-ca-cert" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:09.493832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.493801 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-certs" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.510563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.508255 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f015183f-507c-4318-af33-2190e83755e8-istio-podinfo" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:09.512305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.512276 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-istio-token" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:09.514885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.514834 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-envoy" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.516085 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.515988 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-kube-api-access-dhv7q" pod "f015183f-507c-4318-af33-2190e83755e8" (UID: "f015183f-507c-4318-af33-2190e83755e8"). InnerVolumeSpecName "kube-api-access-dhv7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:09.545301 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.545242 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" podStartSLOduration=2.545228114 podStartE2EDuration="2.545228114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:09.544551637 +0000 UTC m=+2078.225701532" watchObservedRunningTime="2026-10-30 22:17:09.545228114 +0000 UTC m=+2078.226378008" Oct 30 22:17:09.566320 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566258 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566320 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566287 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566320 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566306 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f015183f-507c-4318-af33-2190e83755e8-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566320 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566319 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566333 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566346 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566360 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhv7q\" (UniqueName: \"kubernetes.io/projected/f015183f-507c-4318-af33-2190e83755e8-kube-api-access-dhv7q\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566376 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f015183f-507c-4318-af33-2190e83755e8-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.566653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:09.566390 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f015183f-507c-4318-af33-2190e83755e8-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:10.056017 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.055985 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm"] Oct 30 22:17:10.059692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.059665 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5431d648-5gbfm"] Oct 30 22:17:10.281973 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.281933 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq"] Oct 30 22:17:10.318464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.316528 4933 generic.go:356] "Generic (PLEG): container finished" podID="0c75fcfd-fac3-45d6-a99c-9519c793396e" containerID="2e73c3eb0fc11a2a34ee4f332074416841b63631076ed05e6154ecbbe9fb49c7" exitCode=0 Oct 30 22:17:10.318464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.316732 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" event={"ID":"0c75fcfd-fac3-45d6-a99c-9519c793396e","Type":"ContainerDied","Data":"2e73c3eb0fc11a2a34ee4f332074416841b63631076ed05e6154ecbbe9fb49c7"} Oct 30 22:17:10.350515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.342651 4933 generic.go:356] "Generic (PLEG): container finished" podID="7f58c787-5475-4f57-ae80-2ece9ae38bda" containerID="d4defa57fa7e8ff7815f69c03aecc24e4a0ffdb17e342bc925b22212cf92f1fd" exitCode=0 Oct 30 22:17:10.350515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.343043 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" event={"ID":"7f58c787-5475-4f57-ae80-2ece9ae38bda","Type":"ContainerDied","Data":"d4defa57fa7e8ff7815f69c03aecc24e4a0ffdb17e342bc925b22212cf92f1fd"} Oct 30 22:17:10.378373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.378330 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" event={"ID":"f015183f-507c-4318-af33-2190e83755e8","Type":"ContainerDied","Data":"945ae79ca3aa3660b0426e9b3fa1f7e5e5c10d2f8ba196620e612cc62d9da00b"} Oct 30 22:17:10.378687 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.378667 4933 scope.go:122] "RemoveContainer" containerID="c4c9d425d65eb676a55e977d1e1de52993d9449a59e5f9506108f7b96a6e72ef" Oct 30 22:17:10.399531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.379117 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p" Oct 30 22:17:10.451227 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.426320 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ebddcc-5q7bq"] Oct 30 22:17:10.739852 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.739776 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p"] Oct 30 22:17:10.747361 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:10.747335 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d516c4-openshift-default-6bd584745-cfj6p"] Oct 30 22:17:11.549548 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.545224 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="656c87de-1ed6-4d3f-97db-12c3785e0ed7" path="/var/lib/kubelet/pods/656c87de-1ed6-4d3f-97db-12c3785e0ed7/volumes" Oct 30 22:17:11.549548 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.545981 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3062a4-83a4-4198-928f-6dbc25fad9e0" path="/var/lib/kubelet/pods/ae3062a4-83a4-4198-928f-6dbc25fad9e0/volumes" Oct 30 22:17:11.549548 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.546704 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f015183f-507c-4318-af33-2190e83755e8" path="/var/lib/kubelet/pods/f015183f-507c-4318-af33-2190e83755e8/volumes" Oct 30 22:17:11.644609 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.644523 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s"] Oct 30 22:17:11.727514 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.726731 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b0c55f-j4w8s"] Oct 30 22:17:11.740906 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.740620 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:11.781425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.781401 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:11.823791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.823766 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-xks-ca\") pod \"0c75fcfd-fac3-45d6-a99c-9519c793396e\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " Oct 30 22:17:11.823900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.823808 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-openshift-ca\") pod \"7f58c787-5475-4f57-ae80-2ece9ae38bda\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " Oct 30 22:17:11.823900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.823837 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-openshift-ca\") pod \"0c75fcfd-fac3-45d6-a99c-9519c793396e\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " Oct 30 22:17:11.823900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.823886 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f58c787-5475-4f57-ae80-2ece9ae38bda-kube-api-access-f55hm\" (UniqueName: \"kubernetes.io/projected/7f58c787-5475-4f57-ae80-2ece9ae38bda-kube-api-access-f55hm\") pod \"7f58c787-5475-4f57-ae80-2ece9ae38bda\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " Oct 30 22:17:11.824032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.823910 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c75fcfd-fac3-45d6-a99c-9519c793396e-kube-api-access-7rcm4\" (UniqueName: \"kubernetes.io/projected/0c75fcfd-fac3-45d6-a99c-9519c793396e-kube-api-access-7rcm4\") pod \"0c75fcfd-fac3-45d6-a99c-9519c793396e\" (UID: \"0c75fcfd-fac3-45d6-a99c-9519c793396e\") " Oct 30 22:17:11.824032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.823983 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-xks-ca\") pod \"7f58c787-5475-4f57-ae80-2ece9ae38bda\" (UID: \"7f58c787-5475-4f57-ae80-2ece9ae38bda\") " Oct 30 22:17:11.824190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.824171 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-openshift-ca" pod "7f58c787-5475-4f57-ae80-2ece9ae38bda" (UID: "7f58c787-5475-4f57-ae80-2ece9ae38bda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:11.824276 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.824233 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-openshift-ca" pod "0c75fcfd-fac3-45d6-a99c-9519c793396e" (UID: "0c75fcfd-fac3-45d6-a99c-9519c793396e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:11.824473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.824452 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.824753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.824475 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.826357 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.826330 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c75fcfd-fac3-45d6-a99c-9519c793396e-kube-api-access-7rcm4" pod "0c75fcfd-fac3-45d6-a99c-9519c793396e" (UID: "0c75fcfd-fac3-45d6-a99c-9519c793396e"). InnerVolumeSpecName "kube-api-access-7rcm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.826449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.826375 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f58c787-5475-4f57-ae80-2ece9ae38bda-kube-api-access-f55hm" pod "7f58c787-5475-4f57-ae80-2ece9ae38bda" (UID: "7f58c787-5475-4f57-ae80-2ece9ae38bda"). InnerVolumeSpecName "kube-api-access-f55hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:11.826606 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.826588 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-xks-ca" pod "0c75fcfd-fac3-45d6-a99c-9519c793396e" (UID: "0c75fcfd-fac3-45d6-a99c-9519c793396e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:11.826700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.826682 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-xks-ca" pod "7f58c787-5475-4f57-ae80-2ece9ae38bda" (UID: "7f58c787-5475-4f57-ae80-2ece9ae38bda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:11.925580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.925513 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c75fcfd-fac3-45d6-a99c-9519c793396e-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.925580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.925537 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55hm\" (UniqueName: \"kubernetes.io/projected/7f58c787-5475-4f57-ae80-2ece9ae38bda-kube-api-access-f55hm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.925580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.925548 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rcm4\" (UniqueName: \"kubernetes.io/projected/0c75fcfd-fac3-45d6-a99c-9519c793396e-kube-api-access-7rcm4\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:11.925580 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:11.925557 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f58c787-5475-4f57-ae80-2ece9ae38bda-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:12.395340 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:12.395310 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" Oct 30 22:17:12.395340 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:12.395336 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp" event={"ID":"0c75fcfd-fac3-45d6-a99c-9519c793396e","Type":"ContainerDied","Data":"58d901b3568565fb45d6c8e09d64723448aef1f6740980da74c51a58186e12f6"} Oct 30 22:17:12.395586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:12.395385 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d901b3568565fb45d6c8e09d64723448aef1f6740980da74c51a58186e12f6" Oct 30 22:17:12.396898 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:12.396867 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" Oct 30 22:17:12.397036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:12.396922 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2" event={"ID":"7f58c787-5475-4f57-ae80-2ece9ae38bda","Type":"ContainerDied","Data":"e67f7d5255a9170b1baa4c430cb19f718b7dfa23b129096679e77a45e57c1c84"} Oct 30 22:17:12.397036 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:12.396963 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e67f7d5255a9170b1baa4c430cb19f718b7dfa23b129096679e77a45e57c1c84" Oct 30 22:17:13.465572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:13.465543 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9605d674-61da-49b6-9c86-bfeaf4e3a03f" path="/var/lib/kubelet/pods/9605d674-61da-49b6-9c86-bfeaf4e3a03f/volumes" Oct 30 22:17:18.263898 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:18.263862 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg"] Oct 30 22:17:18.314159 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:18.314131 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-e431fc-lx2vg"] Oct 30 22:17:19.466346 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.466304 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c797faa-b135-4781-a37d-0861c755a8eb" path="/var/lib/kubelet/pods/8c797faa-b135-4781-a37d-0861c755a8eb/volumes" Oct 30 22:17:19.632082 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.632049 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44"] Oct 30 22:17:19.632273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.632234 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" containerID="cri-o://74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68" gracePeriod=30 Oct 30 22:17:19.748984 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.748878 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc"] Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.762843 4933 cpu_manager.go:414] "containerMap: removing container" podUID="7f58c787-5475-4f57-ae80-2ece9ae38bda" containerName="revoke-keys" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.762871 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f58c787-5475-4f57-ae80-2ece9ae38bda" containerName="revoke-keys" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.762924 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0c75fcfd-fac3-45d6-a99c-9519c793396e" containerName="revoke-keys" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.762932 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c75fcfd-fac3-45d6-a99c-9519c793396e" containerName="revoke-keys" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.762948 4933 cpu_manager.go:414] "containerMap: removing container" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.762956 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.763096 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f58c787-5475-4f57-ae80-2ece9ae38bda" containerName="revoke-keys" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.763116 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c75fcfd-fac3-45d6-a99c-9519c793396e" containerName="revoke-keys" Oct 30 22:17:19.765573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.763125 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="f015183f-507c-4318-af33-2190e83755e8" containerName="istio-proxy" Oct 30 22:17:19.772388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.769615 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.814800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.814720 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc"] Oct 30 22:17:19.828532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.826393 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:17:19.828532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.826435 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:19.828532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.826514 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:19.828532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.826606 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:19.840111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.840081 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.840264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.840116 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m95h\" (UniqueName: \"kubernetes.io/projected/efb26b13-cdae-4dfb-b0f5-3b25ed105764-kube-api-access-8m95h\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.840362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.840328 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.941653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.941619 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.941653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.941655 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m95h\" (UniqueName: \"kubernetes.io/projected/efb26b13-cdae-4dfb-b0f5-3b25ed105764-kube-api-access-8m95h\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.941960 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.941786 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.942553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.942531 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:19.944247 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:19.944222 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:20.000320 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:20.000248 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m95h\" (UniqueName: \"kubernetes.io/projected/efb26b13-cdae-4dfb-b0f5-3b25ed105764-kube-api-access-8m95h\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:20.098150 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:20.098126 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:20.481298 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:17:20.481273 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb26b13_cdae_4dfb_b0f5_3b25ed105764.slice/crio-73c1870fb34a18598e1280150df015ab2301353d71d865d12dd2efd657ee31a8 WatchSource:0}: Error finding container 73c1870fb34a18598e1280150df015ab2301353d71d865d12dd2efd657ee31a8: Status 404 returned error can't find the container with id 73c1870fb34a18598e1280150df015ab2301353d71d865d12dd2efd657ee31a8 Oct 30 22:17:20.482194 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:20.482170 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc"] Oct 30 22:17:21.463905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:21.463873 4933 generic.go:356] "Generic (PLEG): container finished" podID="efb26b13-cdae-4dfb-b0f5-3b25ed105764" containerID="abcf0b978f629712a9bf1c5ec1c68555e6ee12d1dc9ece3d4c7f90ebcf2dbde9" exitCode=0 Oct 30 22:17:21.466895 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:21.466866 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" event={"ID":"efb26b13-cdae-4dfb-b0f5-3b25ed105764","Type":"ContainerDied","Data":"abcf0b978f629712a9bf1c5ec1c68555e6ee12d1dc9ece3d4c7f90ebcf2dbde9"} Oct 30 22:17:21.467062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:21.466913 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" event={"ID":"efb26b13-cdae-4dfb-b0f5-3b25ed105764","Type":"ContainerStarted","Data":"73c1870fb34a18598e1280150df015ab2301353d71d865d12dd2efd657ee31a8"} Oct 30 22:17:22.634140 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.634111 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:22.671477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.671450 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb26b13-cdae-4dfb-b0f5-3b25ed105764-kube-api-access-8m95h\" (UniqueName: \"kubernetes.io/projected/efb26b13-cdae-4dfb-b0f5-3b25ed105764-kube-api-access-8m95h\") pod \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " Oct 30 22:17:22.671627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.671584 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-openshift-ca\") pod \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " Oct 30 22:17:22.671694 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.671645 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-xks-ca\") pod \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\" (UID: \"efb26b13-cdae-4dfb-b0f5-3b25ed105764\") " Oct 30 22:17:22.671969 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.671943 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-openshift-ca" pod "efb26b13-cdae-4dfb-b0f5-3b25ed105764" (UID: "efb26b13-cdae-4dfb-b0f5-3b25ed105764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:22.672192 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.672174 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:22.673622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.673603 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb26b13-cdae-4dfb-b0f5-3b25ed105764-kube-api-access-8m95h" pod "efb26b13-cdae-4dfb-b0f5-3b25ed105764" (UID: "efb26b13-cdae-4dfb-b0f5-3b25ed105764"). InnerVolumeSpecName "kube-api-access-8m95h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:22.673789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.673771 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-xks-ca" pod "efb26b13-cdae-4dfb-b0f5-3b25ed105764" (UID: "efb26b13-cdae-4dfb-b0f5-3b25ed105764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:22.773697 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.773612 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb26b13-cdae-4dfb-b0f5-3b25ed105764-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:22.773697 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:22.773637 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m95h\" (UniqueName: \"kubernetes.io/projected/efb26b13-cdae-4dfb-b0f5-3b25ed105764-kube-api-access-8m95h\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:23.484527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:23.481021 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" event={"ID":"efb26b13-cdae-4dfb-b0f5-3b25ed105764","Type":"ContainerDied","Data":"73c1870fb34a18598e1280150df015ab2301353d71d865d12dd2efd657ee31a8"} Oct 30 22:17:23.484527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:23.481145 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73c1870fb34a18598e1280150df015ab2301353d71d865d12dd2efd657ee31a8" Oct 30 22:17:23.484527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:23.481234 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc" Oct 30 22:17:25.339549 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.339526 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 22:17:25.404139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404109 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istiod-ca-cert\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404356 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404157 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-data\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404356 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404183 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-token\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-token\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404356 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404218 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-credential-socket\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404356 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404334 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-envoy\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404370 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-podinfo\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404404 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-socket\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404434 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-data" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:25.404653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404450 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-kube-api-access-lbfsk\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-kube-api-access-lbfsk\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404653 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404536 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-certs\") pod \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\" (UID: \"e067a905-6cfc-41fc-bdea-052a9f35fa8c\") " Oct 30 22:17:25.404945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404673 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-credential-socket" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:25.404945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404710 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istiod-ca-cert" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:25.404945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404806 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-socket" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:25.405114 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.404990 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-certs" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:25.405178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.405117 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.405178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.405136 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.405178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.405152 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.405178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.405172 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.405574 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.405190 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.407029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.406994 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-podinfo" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:25.407029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.407017 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-token" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:25.407204 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.407002 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-envoy" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:25.407204 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.407167 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-kube-api-access-lbfsk" pod "e067a905-6cfc-41fc-bdea-052a9f35fa8c" (UID: "e067a905-6cfc-41fc-bdea-052a9f35fa8c"). InnerVolumeSpecName "kube-api-access-lbfsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:25.499179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.499148 4933 generic.go:356] "Generic (PLEG): container finished" podID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerID="74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68" exitCode=0 Oct 30 22:17:25.499336 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.499193 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" Oct 30 22:17:25.499336 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.499224 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" event={"ID":"e067a905-6cfc-41fc-bdea-052a9f35fa8c","Type":"ContainerDied","Data":"74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68"} Oct 30 22:17:25.499336 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.499274 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44" event={"ID":"e067a905-6cfc-41fc-bdea-052a9f35fa8c","Type":"ContainerDied","Data":"2989bdef8dc2d532d7340dc62ef2c2c9fed7bf39f967b09b11bdb7baf47d3114"} Oct 30 22:17:25.499336 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.499295 4933 scope.go:122] "RemoveContainer" containerID="74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68" Oct 30 22:17:25.506738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.506715 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.506738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.506737 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.506946 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.506747 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e067a905-6cfc-41fc-bdea-052a9f35fa8c-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.506946 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.506757 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfsk\" (UniqueName: \"kubernetes.io/projected/e067a905-6cfc-41fc-bdea-052a9f35fa8c-kube-api-access-lbfsk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:25.510014 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.509936 4933 scope.go:122] "RemoveContainer" containerID="74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68" Oct 30 22:17:25.510243 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:17:25.510219 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68\": container with ID starting with 74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68 not found: ID does not exist" containerID="74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68" Oct 30 22:17:25.510381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.510249 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68"} err="failed to get container status \"74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68\": rpc error: code = NotFound desc = could not find container \"74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68\": container with ID starting with 74d8c2e01066b9fca1510e8c202a65b6178437370a113e5beec63cda0766ed68 not found: ID does not exist" Oct 30 22:17:25.628652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.628622 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44"] Oct 30 22:17:25.635217 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.635186 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b0c55f-openshift-default-85f66ff7f5-wzz44"] Oct 30 22:17:25.701195 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.701170 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs"] Oct 30 22:17:25.701679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.701665 4933 cpu_manager.go:414] "containerMap: removing container" podUID="efb26b13-cdae-4dfb-b0f5-3b25ed105764" containerName="revoke-keys" Oct 30 22:17:25.701740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.701679 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb26b13-cdae-4dfb-b0f5-3b25ed105764" containerName="revoke-keys" Oct 30 22:17:25.701740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.701721 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" Oct 30 22:17:25.701740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.701727 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" Oct 30 22:17:25.701839 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.701800 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" containerName="istio-proxy" Oct 30 22:17:25.701839 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.701810 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb26b13-cdae-4dfb-b0f5-3b25ed105764" containerName="revoke-keys" Oct 30 22:17:25.706384 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.706367 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.759549 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.759460 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:17:25.759692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.759571 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:17:25.759692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.759654 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:17:25.759820 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.759772 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:17:25.760548 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.760527 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs"] Oct 30 22:17:25.809987 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.809959 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.810116 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.810098 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5226g\" (UniqueName: \"kubernetes.io/projected/4365b454-e086-4083-a4e1-7693661b7dd6-kube-api-access-5226g\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.810172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.810151 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.911576 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.911544 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.911718 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.911603 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.911774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.911745 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5226g\" (UniqueName: \"kubernetes.io/projected/4365b454-e086-4083-a4e1-7693661b7dd6-kube-api-access-5226g\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.912399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.912374 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.914141 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.914119 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:25.969710 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:25.969687 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5226g\" (UniqueName: \"kubernetes.io/projected/4365b454-e086-4083-a4e1-7693661b7dd6-kube-api-access-5226g\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:26.019728 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:26.019669 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:26.400109 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:17:26.400082 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4365b454_e086_4083_a4e1_7693661b7dd6.slice/crio-5f18ab305102b80bbe69494e15596beaeb2f695446baba5539feb16556168c71 WatchSource:0}: Error finding container 5f18ab305102b80bbe69494e15596beaeb2f695446baba5539feb16556168c71: Status 404 returned error can't find the container with id 5f18ab305102b80bbe69494e15596beaeb2f695446baba5539feb16556168c71 Oct 30 22:17:26.400554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:26.400278 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs"] Oct 30 22:17:26.509107 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:26.509074 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" event={"ID":"4365b454-e086-4083-a4e1-7693661b7dd6","Type":"ContainerStarted","Data":"5f18ab305102b80bbe69494e15596beaeb2f695446baba5539feb16556168c71"} Oct 30 22:17:27.466038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:27.466006 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e067a905-6cfc-41fc-bdea-052a9f35fa8c" path="/var/lib/kubelet/pods/e067a905-6cfc-41fc-bdea-052a9f35fa8c/volumes" Oct 30 22:17:27.518299 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:27.518072 4933 generic.go:356] "Generic (PLEG): container finished" podID="4365b454-e086-4083-a4e1-7693661b7dd6" containerID="2bc6908114a8f97d508773bc32c30fc1953b7aeccf431e1671146d73d85daad4" exitCode=0 Oct 30 22:17:27.518439 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:27.518266 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" event={"ID":"4365b454-e086-4083-a4e1-7693661b7dd6","Type":"ContainerDied","Data":"2bc6908114a8f97d508773bc32c30fc1953b7aeccf431e1671146d73d85daad4"} Oct 30 22:17:28.548826 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:17:28.548785 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:17:28.699408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.699384 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:28.746628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.746603 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4365b454-e086-4083-a4e1-7693661b7dd6-kube-api-access-5226g\" (UniqueName: \"kubernetes.io/projected/4365b454-e086-4083-a4e1-7693661b7dd6-kube-api-access-5226g\") pod \"4365b454-e086-4083-a4e1-7693661b7dd6\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " Oct 30 22:17:28.746768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.746660 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-openshift-ca\") pod \"4365b454-e086-4083-a4e1-7693661b7dd6\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " Oct 30 22:17:28.746768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.746730 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-xks-ca\") pod \"4365b454-e086-4083-a4e1-7693661b7dd6\" (UID: \"4365b454-e086-4083-a4e1-7693661b7dd6\") " Oct 30 22:17:28.747014 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.746992 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-openshift-ca" pod "4365b454-e086-4083-a4e1-7693661b7dd6" (UID: "4365b454-e086-4083-a4e1-7693661b7dd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:28.747347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.747286 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.748903 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.748881 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4365b454-e086-4083-a4e1-7693661b7dd6-kube-api-access-5226g" pod "4365b454-e086-4083-a4e1-7693661b7dd6" (UID: "4365b454-e086-4083-a4e1-7693661b7dd6"). InnerVolumeSpecName "kube-api-access-5226g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:28.748990 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.748909 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-xks-ca" pod "4365b454-e086-4083-a4e1-7693661b7dd6" (UID: "4365b454-e086-4083-a4e1-7693661b7dd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:28.848272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.848196 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5226g\" (UniqueName: \"kubernetes.io/projected/4365b454-e086-4083-a4e1-7693661b7dd6-kube-api-access-5226g\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:28.848272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:28.848237 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4365b454-e086-4083-a4e1-7693661b7dd6-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:17:29.533789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:29.533762 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" Oct 30 22:17:29.533933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:29.533795 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs" event={"ID":"4365b454-e086-4083-a4e1-7693661b7dd6","Type":"ContainerDied","Data":"5f18ab305102b80bbe69494e15596beaeb2f695446baba5539feb16556168c71"} Oct 30 22:17:29.533933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:29.533833 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f18ab305102b80bbe69494e15596beaeb2f695446baba5539feb16556168c71" Oct 30 22:17:31.903830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:31.903803 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:17:31.905067 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:31.905045 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:17:34.674091 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:34.674057 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9"] Oct 30 22:17:34.722646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:34.722618 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-0a7679-snfr9"] Oct 30 22:17:35.468271 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:35.468235 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddba5b4d-7abb-4e2f-af97-77857ac38088" path="/var/lib/kubelet/pods/ddba5b4d-7abb-4e2f-af97-77857ac38088/volumes" Oct 30 22:17:40.641280 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:40.641211 4933 scope.go:122] "RemoveContainer" containerID="d4f5ec9a84413291bb4bccd02755d49f980b41780fac98ad6495f0878373fb05" Oct 30 22:17:56.674529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:56.674473 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl"] Oct 30 22:17:56.722357 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:56.722324 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-0c99b1-dqhvl"] Oct 30 22:17:57.466374 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:57.466342 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c668cb8b-53e4-42fd-abc5-f5da41cb4260" path="/var/lib/kubelet/pods/c668cb8b-53e4-42fd-abc5-f5da41cb4260/volumes" Oct 30 22:17:57.892230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:57.892199 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc"] Oct 30 22:17:57.942852 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:57.942812 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-db492e-qblxc"] Oct 30 22:17:58.552052 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:17:58.552015 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:17:59.466207 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:17:59.466179 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb26b13-cdae-4dfb-b0f5-3b25ed105764" path="/var/lib/kubelet/pods/efb26b13-cdae-4dfb-b0f5-3b25ed105764/volumes" Oct 30 22:18:03.980768 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:03.980732 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22"] Oct 30 22:18:03.981217 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:03.980951 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" containerID="cri-o://1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef" gracePeriod=30 Oct 30 22:18:04.002293 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:04.002266 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv"] Oct 30 22:18:04.002480 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:04.002458 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" containerID="cri-o://6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026" gracePeriod=30 Oct 30 22:18:04.202542 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:04.202513 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs"] Oct 30 22:18:04.208816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:04.208787 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-5dff71-kmcxs"] Oct 30 22:18:05.466973 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:05.466941 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4365b454-e086-4083-a4e1-7693661b7dd6" path="/var/lib/kubelet/pods/4365b454-e086-4083-a4e1-7693661b7dd6/volumes" Oct 30 22:18:05.950044 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:05.950000 4933 patch_prober.go:29] interesting pod/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:18:05.950213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:05.950073 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:18:05.980516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:05.980456 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:18:06.651342 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.651307 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl"] Oct 30 22:18:06.652061 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.652040 4933 cpu_manager.go:414] "containerMap: removing container" podUID="4365b454-e086-4083-a4e1-7693661b7dd6" containerName="revoke-keys" Oct 30 22:18:06.652165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.652062 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4365b454-e086-4083-a4e1-7693661b7dd6" containerName="revoke-keys" Oct 30 22:18:06.652325 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.652269 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="4365b454-e086-4083-a4e1-7693661b7dd6" containerName="revoke-keys" Oct 30 22:18:06.661360 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.661332 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.709162 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.709131 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.709312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.709178 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qw7b\" (UniqueName: \"kubernetes.io/projected/1740003e-b969-4634-90e0-1ef546ce7d4e-kube-api-access-6qw7b\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.709312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.709292 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.714433 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.714404 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl"] Oct 30 22:18:06.760584 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.760553 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:18:06.760787 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.760726 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:18:06.760787 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.760737 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:18:06.769292 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.769273 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:18:06.810630 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.810606 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.810788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.810640 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qw7b\" (UniqueName: \"kubernetes.io/projected/1740003e-b969-4634-90e0-1ef546ce7d4e-kube-api-access-6qw7b\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.810788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.810712 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.811366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.811332 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.813355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.813334 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.871291 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.871229 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qw7b\" (UniqueName: \"kubernetes.io/projected/1740003e-b969-4634-90e0-1ef546ce7d4e-kube-api-access-6qw7b\") pod \"maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:06.977037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:06.976957 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:07.359137 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:18:07.355558 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1740003e_b969_4634_90e0_1ef546ce7d4e.slice/crio-806883cafcbd8a4527825071977a0605ab0a549f35abfa2a5a2397860e1f3000 WatchSource:0}: Error finding container 806883cafcbd8a4527825071977a0605ab0a549f35abfa2a5a2397860e1f3000: Status 404 returned error can't find the container with id 806883cafcbd8a4527825071977a0605ab0a549f35abfa2a5a2397860e1f3000 Oct 30 22:18:07.359137 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:07.356272 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl"] Oct 30 22:18:07.875029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:07.874996 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" event={"ID":"1740003e-b969-4634-90e0-1ef546ce7d4e","Type":"ContainerStarted","Data":"615e8e67c6d23aad2eec3f371008b73248e5c7e0b27a38bc53cf33a05d0af6e4"} Oct 30 22:18:07.875388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:07.875051 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" event={"ID":"1740003e-b969-4634-90e0-1ef546ce7d4e","Type":"ContainerStarted","Data":"806883cafcbd8a4527825071977a0605ab0a549f35abfa2a5a2397860e1f3000"} Oct 30 22:18:07.994348 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:07.994260 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" podStartSLOduration=1.994245678 podStartE2EDuration="1.994245678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:07.99204296 +0000 UTC m=+2136.673192854" watchObservedRunningTime="2026-10-30 22:18:07.994245678 +0000 UTC m=+2136.675395570" Oct 30 22:18:08.884092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:08.884054 4933 generic.go:356] "Generic (PLEG): container finished" podID="1740003e-b969-4634-90e0-1ef546ce7d4e" containerID="615e8e67c6d23aad2eec3f371008b73248e5c7e0b27a38bc53cf33a05d0af6e4" exitCode=0 Oct 30 22:18:08.884548 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:08.884144 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" event={"ID":"1740003e-b969-4634-90e0-1ef546ce7d4e","Type":"ContainerDied","Data":"615e8e67c6d23aad2eec3f371008b73248e5c7e0b27a38bc53cf33a05d0af6e4"} Oct 30 22:18:09.320048 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.320023 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:18:09.362260 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.362238 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:18:09.447899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.447819 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-data\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.447899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.447862 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-credential-socket\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.448126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.447904 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d841f661-326b-4262-9765-e406ee031b58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d841f661-326b-4262-9765-e406ee031b58-istio-podinfo\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.448126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.447955 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-socket\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.448126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.447990 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d841f661-326b-4262-9765-e406ee031b58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d841f661-326b-4262-9765-e406ee031b58-istiod-ca-cert\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.448126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448022 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-data\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.448126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448065 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f06842ac-3593-45e2-b9e1-e40f53738576-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06842ac-3593-45e2-b9e1-e40f53738576-istio-podinfo\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.448126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448116 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-kube-api-access-wgngk\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-kube-api-access-wgngk\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448117 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-data" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448153 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-certs\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448185 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-envoy\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448234 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-kube-api-access-7fjcz\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-kube-api-access-7fjcz\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448305 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-certs\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448375 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06842ac-3593-45e2-b9e1-e40f53738576-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06842ac-3593-45e2-b9e1-e40f53738576-istiod-ca-cert\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448419 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-envoy\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448422 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d841f661-326b-4262-9765-e406ee031b58-istiod-ca-cert" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:09.448509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448453 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-istio-token\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-istio-token\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.449059 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448690 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-credential-socket" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.449059 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448871 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-socket" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.449059 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.448930 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-istio-token\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-istio-token\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.449059 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.449035 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-credential-socket\") pod \"d841f661-326b-4262-9765-e406ee031b58\" (UID: \"d841f661-326b-4262-9765-e406ee031b58\") " Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.449121 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-socket\") pod \"f06842ac-3593-45e2-b9e1-e40f53738576\" (UID: \"f06842ac-3593-45e2-b9e1-e40f53738576\") " Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.449687 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.449708 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.449721 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.449733 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d841f661-326b-4262-9765-e406ee031b58-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.450198 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-socket" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.450563 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-data" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.450921 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-credential-socket" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.451142 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-certs" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.451406 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06842ac-3593-45e2-b9e1-e40f53738576-istiod-ca-cert" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.452889 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-envoy" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.456723 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.453166 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-certs" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.462313 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.458087 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-kube-api-access-wgngk" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "kube-api-access-wgngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:09.462912 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.462885 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-istio-token" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:09.463038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.462906 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-kube-api-access-7fjcz" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "kube-api-access-7fjcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:09.463038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.462921 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-istio-token" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:09.463038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.463032 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f06842ac-3593-45e2-b9e1-e40f53738576-istio-podinfo" pod "f06842ac-3593-45e2-b9e1-e40f53738576" (UID: "f06842ac-3593-45e2-b9e1-e40f53738576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:09.463200 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.463032 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d841f661-326b-4262-9765-e406ee031b58-istio-podinfo" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:09.464878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.464852 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-envoy" pod "d841f661-326b-4262-9765-e406ee031b58" (UID: "d841f661-326b-4262-9765-e406ee031b58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:09.550890 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.550855 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.550890 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.550892 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.550906 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.550919 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.550932 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d841f661-326b-4262-9765-e406ee031b58-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.550943 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.550983 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06842ac-3593-45e2-b9e1-e40f53738576-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.551003 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgngk\" (UniqueName: \"kubernetes.io/projected/f06842ac-3593-45e2-b9e1-e40f53738576-kube-api-access-wgngk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.551013 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.551024 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.551039 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fjcz\" (UniqueName: \"kubernetes.io/projected/d841f661-326b-4262-9765-e406ee031b58-kube-api-access-7fjcz\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.551050 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06842ac-3593-45e2-b9e1-e40f53738576-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.551059 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06842ac-3593-45e2-b9e1-e40f53738576-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.551092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.551068 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d841f661-326b-4262-9765-e406ee031b58-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:09.899360 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.899316 4933 generic.go:356] "Generic (PLEG): container finished" podID="f06842ac-3593-45e2-b9e1-e40f53738576" containerID="6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026" exitCode=0 Oct 30 22:18:09.899883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.899472 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" event={"ID":"f06842ac-3593-45e2-b9e1-e40f53738576","Type":"ContainerDied","Data":"6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026"} Oct 30 22:18:09.899883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.899602 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" event={"ID":"f06842ac-3593-45e2-b9e1-e40f53738576","Type":"ContainerDied","Data":"cd40eb49253ca764a49980d40a8661ee1144f90f541c75ee5bb39052bca3d8cc"} Oct 30 22:18:09.899883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.899645 4933 scope.go:122] "RemoveContainer" containerID="6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026" Oct 30 22:18:09.900686 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.900336 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv" Oct 30 22:18:09.902539 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.902471 4933 generic.go:356] "Generic (PLEG): container finished" podID="d841f661-326b-4262-9765-e406ee031b58" containerID="1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef" exitCode=0 Oct 30 22:18:09.902664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.902569 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" Oct 30 22:18:09.902664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.902651 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" event={"ID":"d841f661-326b-4262-9765-e406ee031b58","Type":"ContainerDied","Data":"1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef"} Oct 30 22:18:09.902788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.902694 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22" event={"ID":"d841f661-326b-4262-9765-e406ee031b58","Type":"ContainerDied","Data":"6661a20fff01bc0ecf44d92dfe6fcfcabde393eb5dde93821575529a52d19438"} Oct 30 22:18:09.915033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.915016 4933 scope.go:122] "RemoveContainer" containerID="6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026" Oct 30 22:18:09.915333 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:09.915314 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026\": container with ID starting with 6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026 not found: ID does not exist" containerID="6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026" Oct 30 22:18:09.915417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.915340 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026"} err="failed to get container status \"6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026\": rpc error: code = NotFound desc = could not find container \"6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026\": container with ID starting with 6a606fae7443bc61a830fd4058cc68c24040c168d3bbcf8c4e0fac9f8f67a026 not found: ID does not exist" Oct 30 22:18:09.915417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.915356 4933 scope.go:122] "RemoveContainer" containerID="1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef" Oct 30 22:18:09.925758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.925742 4933 scope.go:122] "RemoveContainer" containerID="1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef" Oct 30 22:18:09.926523 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:09.926465 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef\": container with ID starting with 1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef not found: ID does not exist" containerID="1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef" Oct 30 22:18:09.931105 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:09.931065 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef"} err="failed to get container status \"1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef\": rpc error: code = NotFound desc = could not find container \"1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef\": container with ID starting with 1b5882a8366f3efbd6d96baa6c55173ac0acfe5e41c2bb25a4e073d977e1f1ef not found: ID does not exist" Oct 30 22:18:10.057472 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.057442 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:10.071155 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.071129 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv"] Oct 30 22:18:10.079950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.079928 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-987mv"] Oct 30 22:18:10.157459 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.157391 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-openshift-ca\") pod \"1740003e-b969-4634-90e0-1ef546ce7d4e\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " Oct 30 22:18:10.157620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.157525 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1740003e-b969-4634-90e0-1ef546ce7d4e-kube-api-access-6qw7b\" (UniqueName: \"kubernetes.io/projected/1740003e-b969-4634-90e0-1ef546ce7d4e-kube-api-access-6qw7b\") pod \"1740003e-b969-4634-90e0-1ef546ce7d4e\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " Oct 30 22:18:10.157695 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.157621 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-xks-ca\") pod \"1740003e-b969-4634-90e0-1ef546ce7d4e\" (UID: \"1740003e-b969-4634-90e0-1ef546ce7d4e\") " Oct 30 22:18:10.157780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.157757 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-openshift-ca" pod "1740003e-b969-4634-90e0-1ef546ce7d4e" (UID: "1740003e-b969-4634-90e0-1ef546ce7d4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:10.158006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.157989 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:10.159808 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.159783 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1740003e-b969-4634-90e0-1ef546ce7d4e-kube-api-access-6qw7b" pod "1740003e-b969-4634-90e0-1ef546ce7d4e" (UID: "1740003e-b969-4634-90e0-1ef546ce7d4e"). InnerVolumeSpecName "kube-api-access-6qw7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:10.159900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.159828 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-xks-ca" pod "1740003e-b969-4634-90e0-1ef546ce7d4e" (UID: "1740003e-b969-4634-90e0-1ef546ce7d4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:10.194885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.194861 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22"] Oct 30 22:18:10.199159 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.199135 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0c99b1-openshift-default-967855977-pnm22"] Oct 30 22:18:10.259370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.259346 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1740003e-b969-4634-90e0-1ef546ce7d4e-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:10.259370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.259371 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qw7b\" (UniqueName: \"kubernetes.io/projected/1740003e-b969-4634-90e0-1ef546ce7d4e-kube-api-access-6qw7b\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:10.808527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.808475 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg"] Oct 30 22:18:10.808769 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.808740 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" containerID="cri-o://a0a64024af02b745e325f1464c38e70f9fb56a94ff30d95bee61deebc79c35d9" gracePeriod=30 Oct 30 22:18:10.956802 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.956767 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" event={"ID":"1740003e-b969-4634-90e0-1ef546ce7d4e","Type":"ContainerDied","Data":"806883cafcbd8a4527825071977a0605ab0a549f35abfa2a5a2397860e1f3000"} Oct 30 22:18:10.957614 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.957574 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="806883cafcbd8a4527825071977a0605ab0a549f35abfa2a5a2397860e1f3000" Oct 30 22:18:10.957788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:10.957077 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl" Oct 30 22:18:11.151894 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.151813 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz"] Oct 30 22:18:11.267151 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.267117 4933 patch_prober.go:29] interesting pod/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:18:11.267327 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.267172 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:18:11.469814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.469737 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d841f661-326b-4262-9765-e406ee031b58" path="/var/lib/kubelet/pods/d841f661-326b-4262-9765-e406ee031b58/volumes" Oct 30 22:18:11.470381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.470361 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" path="/var/lib/kubelet/pods/f06842ac-3593-45e2-b9e1-e40f53738576/volumes" Oct 30 22:18:11.803045 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803014 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c"] Oct 30 22:18:11.803602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803584 4933 cpu_manager.go:414] "containerMap: removing container" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" Oct 30 22:18:11.803737 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803604 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" Oct 30 22:18:11.803737 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803655 4933 cpu_manager.go:414] "containerMap: removing container" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" Oct 30 22:18:11.803737 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803663 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" Oct 30 22:18:11.803737 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803674 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1740003e-b969-4634-90e0-1ef546ce7d4e" containerName="revoke-keys" Oct 30 22:18:11.803737 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803681 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1740003e-b969-4634-90e0-1ef546ce7d4e" containerName="revoke-keys" Oct 30 22:18:11.804015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803825 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="d841f661-326b-4262-9765-e406ee031b58" containerName="istio-proxy" Oct 30 22:18:11.804015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803840 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1740003e-b969-4634-90e0-1ef546ce7d4e" containerName="revoke-keys" Oct 30 22:18:11.804015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.803857 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06842ac-3593-45e2-b9e1-e40f53738576" containerName="istio-proxy" Oct 30 22:18:11.808553 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.808534 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.862379 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.862356 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:18:11.862593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.862414 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:18:11.863209 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.863189 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:18:11.866077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.866052 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c"] Oct 30 22:18:11.878818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.878792 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.879023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.878999 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.879074 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.879045 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrvf\" (UniqueName: \"kubernetes.io/projected/23284b96-dc1f-4cd1-a1c9-affdc073a307-kube-api-access-4lrvf\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.911039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.911015 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:18:11.980709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.980678 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.981143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.980815 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.981143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.980856 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrvf\" (UniqueName: \"kubernetes.io/projected/23284b96-dc1f-4cd1-a1c9-affdc073a307-kube-api-access-4lrvf\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.981594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.981574 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:11.983148 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:11.983129 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:12.039172 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:12.039139 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrvf\" (UniqueName: \"kubernetes.io/projected/23284b96-dc1f-4cd1-a1c9-affdc073a307-kube-api-access-4lrvf\") pod \"maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:12.121547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:12.121469 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:12.503953 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:18:12.503926 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23284b96_dc1f_4cd1_a1c9_affdc073a307.slice/crio-618e3a91e9173a31cd71b75094146a88d3e5932a579a64d8983737730fed49a9 WatchSource:0}: Error finding container 618e3a91e9173a31cd71b75094146a88d3e5932a579a64d8983737730fed49a9: Status 404 returned error can't find the container with id 618e3a91e9173a31cd71b75094146a88d3e5932a579a64d8983737730fed49a9 Oct 30 22:18:12.504069 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:12.503995 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c"] Oct 30 22:18:12.982558 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:12.982521 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" event={"ID":"23284b96-dc1f-4cd1-a1c9-affdc073a307","Type":"ContainerStarted","Data":"6dcdbfdd56429346bf07160a56cced34eeae2342d8deb11c552f0c34130555f4"} Oct 30 22:18:12.983009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:12.982603 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" event={"ID":"23284b96-dc1f-4cd1-a1c9-affdc073a307","Type":"ContainerStarted","Data":"618e3a91e9173a31cd71b75094146a88d3e5932a579a64d8983737730fed49a9"} Oct 30 22:18:13.104476 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:13.104427 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" podStartSLOduration=2.104402765 podStartE2EDuration="2.104402765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:13.100594514 +0000 UTC m=+2141.781744403" watchObservedRunningTime="2026-10-30 22:18:13.104402765 +0000 UTC m=+2141.785552661" Oct 30 22:18:13.990966 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:13.990934 4933 generic.go:356] "Generic (PLEG): container finished" podID="23284b96-dc1f-4cd1-a1c9-affdc073a307" containerID="6dcdbfdd56429346bf07160a56cced34eeae2342d8deb11c552f0c34130555f4" exitCode=0 Oct 30 22:18:13.991374 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:13.991025 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" event={"ID":"23284b96-dc1f-4cd1-a1c9-affdc073a307","Type":"ContainerDied","Data":"6dcdbfdd56429346bf07160a56cced34eeae2342d8deb11c552f0c34130555f4"} Oct 30 22:18:15.146856 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.146826 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:15.217933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.217911 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-openshift-ca\") pod \"23284b96-dc1f-4cd1-a1c9-affdc073a307\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " Oct 30 22:18:15.218081 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.218028 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-xks-ca\") pod \"23284b96-dc1f-4cd1-a1c9-affdc073a307\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " Oct 30 22:18:15.218149 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.218097 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23284b96-dc1f-4cd1-a1c9-affdc073a307-kube-api-access-4lrvf\" (UniqueName: \"kubernetes.io/projected/23284b96-dc1f-4cd1-a1c9-affdc073a307-kube-api-access-4lrvf\") pod \"23284b96-dc1f-4cd1-a1c9-affdc073a307\" (UID: \"23284b96-dc1f-4cd1-a1c9-affdc073a307\") " Oct 30 22:18:15.218307 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.218283 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-openshift-ca" pod "23284b96-dc1f-4cd1-a1c9-affdc073a307" (UID: "23284b96-dc1f-4cd1-a1c9-affdc073a307"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:15.218448 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.218430 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:15.220244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.220225 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-xks-ca" pod "23284b96-dc1f-4cd1-a1c9-affdc073a307" (UID: "23284b96-dc1f-4cd1-a1c9-affdc073a307"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:15.220296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.220250 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23284b96-dc1f-4cd1-a1c9-affdc073a307-kube-api-access-4lrvf" pod "23284b96-dc1f-4cd1-a1c9-affdc073a307" (UID: "23284b96-dc1f-4cd1-a1c9-affdc073a307"). InnerVolumeSpecName "kube-api-access-4lrvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:15.319541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.319515 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23284b96-dc1f-4cd1-a1c9-affdc073a307-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:15.319664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:15.319542 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrvf\" (UniqueName: \"kubernetes.io/projected/23284b96-dc1f-4cd1-a1c9-affdc073a307-kube-api-access-4lrvf\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.011704 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.011678 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" Oct 30 22:18:16.011872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.011720 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c" event={"ID":"23284b96-dc1f-4cd1-a1c9-affdc073a307","Type":"ContainerDied","Data":"618e3a91e9173a31cd71b75094146a88d3e5932a579a64d8983737730fed49a9"} Oct 30 22:18:16.011872 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.011777 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="618e3a91e9173a31cd71b75094146a88d3e5932a579a64d8983737730fed49a9" Oct 30 22:18:16.014242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.014215 4933 generic.go:356] "Generic (PLEG): container finished" podID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerID="a0a64024af02b745e325f1464c38e70f9fb56a94ff30d95bee61deebc79c35d9" exitCode=0 Oct 30 22:18:16.014387 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.014292 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" event={"ID":"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd","Type":"ContainerDied","Data":"a0a64024af02b745e325f1464c38e70f9fb56a94ff30d95bee61deebc79c35d9"} Oct 30 22:18:16.068290 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.068261 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:18:16.230268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230240 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-socket\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230294 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-podinfo\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230351 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-data\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230387 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-token\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-token\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230483 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-credential-socket\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230547 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-certs\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230587 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istiod-ca-cert\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230632 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-envoy\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.230789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.230684 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-kube-api-access-zk7jq\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-kube-api-access-zk7jq\") pod \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\" (UID: \"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd\") " Oct 30 22:18:16.231203 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.231145 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-certs" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.231266 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.231231 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istiod-ca-cert" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:16.231446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.231419 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-credential-socket" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.231742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.231709 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-data" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.232325 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.231898 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-socket" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.233035 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.233009 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx"] Oct 30 22:18:16.234891 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.234861 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-kube-api-access-zk7jq" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "kube-api-access-zk7jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:16.236677 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.236657 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-envoy" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:16.239682 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.239643 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-token" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:16.252861 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.252838 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-podinfo" pod "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" (UID: "48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:16.256394 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.256371 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb"] Oct 30 22:18:16.332622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332557 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332587 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332601 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332614 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332628 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk7jq\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-kube-api-access-zk7jq\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332979 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332640 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332979 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332651 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332979 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332663 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.332979 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.332673 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:16.969307 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.969277 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv"] Oct 30 22:18:16.969859 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:16.969826 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" containerID="cri-o://8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225" gracePeriod=30 Oct 30 22:18:17.023501 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:17.023472 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" Oct 30 22:18:17.023678 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:17.023529 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg" event={"ID":"48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd","Type":"ContainerDied","Data":"22c62a1d121c0a40fd688ccdf3b1a81bf9d30fd013e7911de0d345ae815831e8"} Oct 30 22:18:17.023678 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:17.023605 4933 scope.go:122] "RemoveContainer" containerID="a0a64024af02b745e325f1464c38e70f9fb56a94ff30d95bee61deebc79c35d9" Oct 30 22:18:17.244529 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:17.244431 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg"] Oct 30 22:18:17.262212 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:17.262177 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-5dff71-openshift-default-79bd87bcfc-wxdkg"] Oct 30 22:18:17.467039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:17.467003 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" path="/var/lib/kubelet/pods/48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd/volumes" Oct 30 22:18:19.118831 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:19.118794 4933 patch_prober.go:29] interesting pod/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:18:19.119244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:19.118863 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:18:22.252093 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.252038 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:18:22.402865 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.402837 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-podinfo\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.402881 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-envoy\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.402908 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-socket\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.402950 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-certs\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.402982 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-token\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-token\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403050 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403032 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e722ade8-3623-4a25-af85-19ba70f18ab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e722ade8-3623-4a25-af85-19ba70f18ab0-istiod-ca-cert\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403160 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-credential-socket\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403214 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-kube-api-access-srl4n\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-kube-api-access-srl4n\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403266 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-certs" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:22.403341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403319 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-data\") pod \"e722ade8-3623-4a25-af85-19ba70f18ab0\" (UID: \"e722ade8-3623-4a25-af85-19ba70f18ab0\") " Oct 30 22:18:22.403341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403298 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-socket" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:22.403819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403431 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-credential-socket" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:22.403819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403440 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e722ade8-3623-4a25-af85-19ba70f18ab0-istiod-ca-cert" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:22.403819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403618 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-data" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:22.403819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403786 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.403819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403801 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.403819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403816 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e722ade8-3623-4a25-af85-19ba70f18ab0-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.403819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403825 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.404199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.403834 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.405367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.405342 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-envoy" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:22.405511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.405386 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-token" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:22.405511 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.405448 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-podinfo" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:18:22.405828 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.405807 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-kube-api-access-srl4n" pod "e722ade8-3623-4a25-af85-19ba70f18ab0" (UID: "e722ade8-3623-4a25-af85-19ba70f18ab0"). InnerVolumeSpecName "kube-api-access-srl4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:22.505334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.505256 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srl4n\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-kube-api-access-srl4n\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.505334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.505286 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.505334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.505296 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:22.505334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:22.505305 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e722ade8-3623-4a25-af85-19ba70f18ab0-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:23.085807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.085775 4933 generic.go:356] "Generic (PLEG): container finished" podID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerID="8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225" exitCode=0 Oct 30 22:18:23.085989 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.085819 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" Oct 30 22:18:23.085989 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.085851 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" event={"ID":"e722ade8-3623-4a25-af85-19ba70f18ab0","Type":"ContainerDied","Data":"8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225"} Oct 30 22:18:23.085989 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.085907 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv" event={"ID":"e722ade8-3623-4a25-af85-19ba70f18ab0","Type":"ContainerDied","Data":"a3fe4106fb33a380f6a04a1c8fa96ba46cd07f93b246944f113b1aa523e0a985"} Oct 30 22:18:23.085989 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.085931 4933 scope.go:122] "RemoveContainer" containerID="8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225" Oct 30 22:18:23.097278 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.097135 4933 scope.go:122] "RemoveContainer" containerID="8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225" Oct 30 22:18:23.097367 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:23.097351 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225\": container with ID starting with 8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225 not found: ID does not exist" containerID="8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225" Oct 30 22:18:23.097406 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.097376 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225"} err="failed to get container status \"8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225\": rpc error: code = NotFound desc = could not find container \"8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225\": container with ID starting with 8f19738d69e530b40dd7652063e187f1cdc5694b5c5c5355ddb76af9119a5225 not found: ID does not exist" Oct 30 22:18:23.258586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.258555 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv"] Oct 30 22:18:23.308264 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.308236 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-zbvlv"] Oct 30 22:18:23.311191 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.311171 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm"] Oct 30 22:18:23.466272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:23.466184 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" path="/var/lib/kubelet/pods/e722ade8-3623-4a25-af85-19ba70f18ab0/volumes" Oct 30 22:18:26.173715 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.173677 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" containerID="cri-o://54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d" gracePeriod=45 Oct 30 22:18:26.428400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.428328 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:18:26.451538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451510 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-home\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-home\") pod \"036b01a5-dc67-4162-8c8a-440786cbe489\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " Oct 30 22:18:26.451650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451557 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/036b01a5-dc67-4162-8c8a-440786cbe489-tls-certs\" (UniqueName: \"kubernetes.io/secret/036b01a5-dc67-4162-8c8a-440786cbe489-tls-certs\") pod \"036b01a5-dc67-4162-8c8a-440786cbe489\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " Oct 30 22:18:26.451650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451593 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-tmp-dir\") pod \"036b01a5-dc67-4162-8c8a-440786cbe489\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " Oct 30 22:18:26.451779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451655 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-model-cache\") pod \"036b01a5-dc67-4162-8c8a-440786cbe489\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " Oct 30 22:18:26.451779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451697 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036b01a5-dc67-4162-8c8a-440786cbe489-kube-api-access-fgfkj\" (UniqueName: \"kubernetes.io/projected/036b01a5-dc67-4162-8c8a-440786cbe489-kube-api-access-fgfkj\") pod \"036b01a5-dc67-4162-8c8a-440786cbe489\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " Oct 30 22:18:26.451779 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451760 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-home" pod "036b01a5-dc67-4162-8c8a-440786cbe489" (UID: "036b01a5-dc67-4162-8c8a-440786cbe489"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.451934 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451788 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-dshm\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-dshm\") pod \"036b01a5-dc67-4162-8c8a-440786cbe489\" (UID: \"036b01a5-dc67-4162-8c8a-440786cbe489\") " Oct 30 22:18:26.451995 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451937 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-tmp-dir" pod "036b01a5-dc67-4162-8c8a-440786cbe489" (UID: "036b01a5-dc67-4162-8c8a-440786cbe489"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.451995 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.451975 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-model-cache" pod "036b01a5-dc67-4162-8c8a-440786cbe489" (UID: "036b01a5-dc67-4162-8c8a-440786cbe489"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.452192 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.452175 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.452284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.452198 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.452284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.452214 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.454048 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.454025 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/036b01a5-dc67-4162-8c8a-440786cbe489-tls-certs" pod "036b01a5-dc67-4162-8c8a-440786cbe489" (UID: "036b01a5-dc67-4162-8c8a-440786cbe489"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:26.454227 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.454063 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-dshm" pod "036b01a5-dc67-4162-8c8a-440786cbe489" (UID: "036b01a5-dc67-4162-8c8a-440786cbe489"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:26.454227 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.454144 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036b01a5-dc67-4162-8c8a-440786cbe489-kube-api-access-fgfkj" pod "036b01a5-dc67-4162-8c8a-440786cbe489" (UID: "036b01a5-dc67-4162-8c8a-440786cbe489"). InnerVolumeSpecName "kube-api-access-fgfkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:26.553966 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.553937 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/036b01a5-dc67-4162-8c8a-440786cbe489-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.553966 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.553966 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/036b01a5-dc67-4162-8c8a-440786cbe489-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:26.553966 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:26.553981 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfkj\" (UniqueName: \"kubernetes.io/projected/036b01a5-dc67-4162-8c8a-440786cbe489-kube-api-access-fgfkj\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:27.116164 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.116123 4933 generic.go:356] "Generic (PLEG): container finished" podID="036b01a5-dc67-4162-8c8a-440786cbe489" containerID="54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d" exitCode=0 Oct 30 22:18:27.116415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.116172 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" Oct 30 22:18:27.116415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.116207 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" event={"ID":"036b01a5-dc67-4162-8c8a-440786cbe489","Type":"ContainerDied","Data":"54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d"} Oct 30 22:18:27.116415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.116248 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz" event={"ID":"036b01a5-dc67-4162-8c8a-440786cbe489","Type":"ContainerDied","Data":"cfaabb192434f4fe3182689ab00093d03c542c9467c924e7f137cb8b786aaba0"} Oct 30 22:18:27.116415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.116270 4933 scope.go:122] "RemoveContainer" containerID="54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d" Oct 30 22:18:27.135157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.135136 4933 scope.go:122] "RemoveContainer" containerID="54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d" Oct 30 22:18:27.135449 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:27.135424 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d\": container with ID starting with 54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d not found: ID does not exist" containerID="54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d" Oct 30 22:18:27.135586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.135456 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d"} err="failed to get container status \"54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d\": rpc error: code = NotFound desc = could not find container \"54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d\": container with ID starting with 54175cd5b3f3b71358625dc436a70b6e4b7bbf042cda75ea29fca1d16aba005d not found: ID does not exist" Oct 30 22:18:27.285156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.285083 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz"] Oct 30 22:18:27.335460 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.335432 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-dde1e8/simulator-w0-dde1e8-kserve-69cc64f68b-fcksz"] Oct 30 22:18:27.466626 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:27.466550 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" path="/var/lib/kubelet/pods/036b01a5-dc67-4162-8c8a-440786cbe489/volumes" Oct 30 22:18:28.311638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:28.311566 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" podUID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" containerName="payload-processing" containerID="cri-o://08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8" gracePeriod=25 Oct 30 22:18:28.543501 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:28.543454 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:18:31.091250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.091208 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" podUID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:18:31.263639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.263580 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" containerID="cri-o://7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50" gracePeriod=45 Oct 30 22:18:31.277921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.277886 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" containerID="cri-o://d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3" gracePeriod=45 Oct 30 22:18:31.344404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.344329 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": dial tcp 10.132.0.103:8000: connect: connection refused" Oct 30 22:18:31.546910 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.546880 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:18:31.593840 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.593817 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:18:31.604388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604327 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d61953-e352-44cd-9276-8285a2e665dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/91d61953-e352-44cd-9276-8285a2e665dc-tls-certs\") pod \"91d61953-e352-44cd-9276-8285a2e665dc\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " Oct 30 22:18:31.604524 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604396 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-home\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-home\") pod \"91d61953-e352-44cd-9276-8285a2e665dc\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " Oct 30 22:18:31.604524 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604508 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-model-cache\") pod \"91d61953-e352-44cd-9276-8285a2e665dc\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " Oct 30 22:18:31.604650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604555 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d61953-e352-44cd-9276-8285a2e665dc-kube-api-access-pm6xj\" (UniqueName: \"kubernetes.io/projected/91d61953-e352-44cd-9276-8285a2e665dc-kube-api-access-pm6xj\") pod \"91d61953-e352-44cd-9276-8285a2e665dc\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " Oct 30 22:18:31.604650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604602 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-dshm\") pod \"91d61953-e352-44cd-9276-8285a2e665dc\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " Oct 30 22:18:31.604650 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604636 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-tmp-dir\") pod \"91d61953-e352-44cd-9276-8285a2e665dc\" (UID: \"91d61953-e352-44cd-9276-8285a2e665dc\") " Oct 30 22:18:31.604807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604657 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-home" pod "91d61953-e352-44cd-9276-8285a2e665dc" (UID: "91d61953-e352-44cd-9276-8285a2e665dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.604807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604755 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-model-cache" pod "91d61953-e352-44cd-9276-8285a2e665dc" (UID: "91d61953-e352-44cd-9276-8285a2e665dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.604909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.604877 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-tmp-dir" pod "91d61953-e352-44cd-9276-8285a2e665dc" (UID: "91d61953-e352-44cd-9276-8285a2e665dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.605125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.605108 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.605445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.605127 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.605445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.605136 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.606758 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.606733 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-dshm" pod "91d61953-e352-44cd-9276-8285a2e665dc" (UID: "91d61953-e352-44cd-9276-8285a2e665dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.606847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.606778 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d61953-e352-44cd-9276-8285a2e665dc-tls-certs" pod "91d61953-e352-44cd-9276-8285a2e665dc" (UID: "91d61953-e352-44cd-9276-8285a2e665dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:31.606847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.606823 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d61953-e352-44cd-9276-8285a2e665dc-kube-api-access-pm6xj" pod "91d61953-e352-44cd-9276-8285a2e665dc" (UID: "91d61953-e352-44cd-9276-8285a2e665dc"). InnerVolumeSpecName "kube-api-access-pm6xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:31.705509 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.705461 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-dshm\") pod \"c201be91-d35d-4319-9fea-01c061905c27\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " Oct 30 22:18:31.705688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.705576 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-home\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-home\") pod \"c201be91-d35d-4319-9fea-01c061905c27\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " Oct 30 22:18:31.705688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.705616 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c201be91-d35d-4319-9fea-01c061905c27-tls-certs\" (UniqueName: \"kubernetes.io/secret/c201be91-d35d-4319-9fea-01c061905c27-tls-certs\") pod \"c201be91-d35d-4319-9fea-01c061905c27\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " Oct 30 22:18:31.705688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.705681 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c201be91-d35d-4319-9fea-01c061905c27-kube-api-access-jrmsq\" (UniqueName: \"kubernetes.io/projected/c201be91-d35d-4319-9fea-01c061905c27-kube-api-access-jrmsq\") pod \"c201be91-d35d-4319-9fea-01c061905c27\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " Oct 30 22:18:31.705876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.705717 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-model-cache\") pod \"c201be91-d35d-4319-9fea-01c061905c27\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " Oct 30 22:18:31.705876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.705815 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-tmp-dir\") pod \"c201be91-d35d-4319-9fea-01c061905c27\" (UID: \"c201be91-d35d-4319-9fea-01c061905c27\") " Oct 30 22:18:31.706001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.705863 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-home" pod "c201be91-d35d-4319-9fea-01c061905c27" (UID: "c201be91-d35d-4319-9fea-01c061905c27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.706355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.706172 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.706355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.706192 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d61953-e352-44cd-9276-8285a2e665dc-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.706355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.706208 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6xj\" (UniqueName: \"kubernetes.io/projected/91d61953-e352-44cd-9276-8285a2e665dc-kube-api-access-pm6xj\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.706355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.706221 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d61953-e352-44cd-9276-8285a2e665dc-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.706355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.706322 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-model-cache" pod "c201be91-d35d-4319-9fea-01c061905c27" (UID: "c201be91-d35d-4319-9fea-01c061905c27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.706756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.706429 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-tmp-dir" pod "c201be91-d35d-4319-9fea-01c061905c27" (UID: "c201be91-d35d-4319-9fea-01c061905c27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.708126 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.708092 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c201be91-d35d-4319-9fea-01c061905c27-kube-api-access-jrmsq" pod "c201be91-d35d-4319-9fea-01c061905c27" (UID: "c201be91-d35d-4319-9fea-01c061905c27"). InnerVolumeSpecName "kube-api-access-jrmsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:31.708365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.708337 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-dshm" pod "c201be91-d35d-4319-9fea-01c061905c27" (UID: "c201be91-d35d-4319-9fea-01c061905c27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:18:31.708445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.708372 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c201be91-d35d-4319-9fea-01c061905c27-tls-certs" pod "c201be91-d35d-4319-9fea-01c061905c27" (UID: "c201be91-d35d-4319-9fea-01c061905c27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:31.807371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.807332 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrmsq\" (UniqueName: \"kubernetes.io/projected/c201be91-d35d-4319-9fea-01c061905c27-kube-api-access-jrmsq\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.807371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.807372 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.807657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.807387 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.807657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.807402 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c201be91-d35d-4319-9fea-01c061905c27-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:31.807657 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:31.807415 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c201be91-d35d-4319-9fea-01c061905c27-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:32.158426 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.158392 4933 generic.go:356] "Generic (PLEG): container finished" podID="c201be91-d35d-4319-9fea-01c061905c27" containerID="d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3" exitCode=0 Oct 30 22:18:32.158937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.158448 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" Oct 30 22:18:32.158937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.158478 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" event={"ID":"c201be91-d35d-4319-9fea-01c061905c27","Type":"ContainerDied","Data":"d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3"} Oct 30 22:18:32.158937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.158573 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb" event={"ID":"c201be91-d35d-4319-9fea-01c061905c27","Type":"ContainerDied","Data":"040e7880d4d7803c0c967d03ca0f7c3c28eb25faab36d1e7b4df967707aaf97f"} Oct 30 22:18:32.158937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.158601 4933 scope.go:122] "RemoveContainer" containerID="d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3" Oct 30 22:18:32.160131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.160109 4933 generic.go:356] "Generic (PLEG): container finished" podID="91d61953-e352-44cd-9276-8285a2e665dc" containerID="7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50" exitCode=0 Oct 30 22:18:32.160232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.160157 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" Oct 30 22:18:32.160232 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.160193 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" event={"ID":"91d61953-e352-44cd-9276-8285a2e665dc","Type":"ContainerDied","Data":"7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50"} Oct 30 22:18:32.160343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.160226 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx" event={"ID":"91d61953-e352-44cd-9276-8285a2e665dc","Type":"ContainerDied","Data":"b7b2916fff8dc6057453173e350a1a0da15289c33ce62ae477af83f7efece7e2"} Oct 30 22:18:32.170248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.170231 4933 scope.go:122] "RemoveContainer" containerID="d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3" Oct 30 22:18:32.170542 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:32.170522 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3\": container with ID starting with d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3 not found: ID does not exist" containerID="d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3" Oct 30 22:18:32.170665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.170548 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3"} err="failed to get container status \"d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3\": rpc error: code = NotFound desc = could not find container \"d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3\": container with ID starting with d86b2d072f068041a0f11c76eff5bba74bb2f85ab7b44748c3b416d48d8eeae3 not found: ID does not exist" Oct 30 22:18:32.170665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.170564 4933 scope.go:122] "RemoveContainer" containerID="7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50" Oct 30 22:18:32.180190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.180168 4933 scope.go:122] "RemoveContainer" containerID="7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50" Oct 30 22:18:32.180505 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:32.180463 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50\": container with ID starting with 7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50 not found: ID does not exist" containerID="7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50" Oct 30 22:18:32.180592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.180509 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50"} err="failed to get container status \"7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50\": rpc error: code = NotFound desc = could not find container \"7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50\": container with ID starting with 7b6fc1cff990bc8ea2d045c781cb8c371946f4c4ae069205dae3b81cf9ea1e50 not found: ID does not exist" Oct 30 22:18:32.327889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.327866 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx"] Oct 30 22:18:32.330780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.330748 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/distinct-2-w5-9f8aea-kserve-68bc5ddc7-j6wdx"] Oct 30 22:18:32.440786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.440704 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb"] Oct 30 22:18:32.443658 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:32.443635 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9f8aea/simulator-w5-9f8aea-kserve-56ddfcb466-scdxb"] Oct 30 22:18:33.138295 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138262 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf"] Oct 30 22:18:33.138747 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138733 4933 cpu_manager.go:414] "containerMap: removing container" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" Oct 30 22:18:33.138800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138749 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" Oct 30 22:18:33.138800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138766 4933 cpu_manager.go:414] "containerMap: removing container" podUID="23284b96-dc1f-4cd1-a1c9-affdc073a307" containerName="revoke-keys" Oct 30 22:18:33.138800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138772 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23284b96-dc1f-4cd1-a1c9-affdc073a307" containerName="revoke-keys" Oct 30 22:18:33.138800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138786 4933 cpu_manager.go:414] "containerMap: removing container" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" Oct 30 22:18:33.138800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138791 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" Oct 30 22:18:33.138800 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138801 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138805 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138814 4933 cpu_manager.go:414] "containerMap: removing container" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138818 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138824 4933 cpu_manager.go:414] "containerMap: removing container" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138829 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138937 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d61953-e352-44cd-9276-8285a2e665dc" containerName="main" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138949 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="23284b96-dc1f-4cd1-a1c9-affdc073a307" containerName="revoke-keys" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138956 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="c201be91-d35d-4319-9fea-01c061905c27" containerName="main" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138961 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e722ade8-3623-4a25-af85-19ba70f18ab0" containerName="istio-proxy" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138966 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="036b01a5-dc67-4162-8c8a-440786cbe489" containerName="main" Oct 30 22:18:33.139242 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.138972 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="48074c0f-ddd9-4a6f-ac8a-c4397b7db0cd" containerName="istio-proxy" Oct 30 22:18:33.143572 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.143550 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.198171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.198046 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:18:33.198171 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.198127 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:18:33.201350 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.201325 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf"] Oct 30 22:18:33.219563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.219524 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4whdv\" (UniqueName: \"kubernetes.io/projected/74d405d8-b58c-4358-be4d-f5a8276edb48-kube-api-access-4whdv\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.219709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.219596 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.219807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.219779 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.246334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.246312 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:18:33.246334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.246321 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:18:33.320838 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.320802 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.321002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.320921 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4whdv\" (UniqueName: \"kubernetes.io/projected/74d405d8-b58c-4358-be4d-f5a8276edb48-kube-api-access-4whdv\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.321002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.320978 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.321919 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.321892 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.323937 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.323912 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.397512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.397430 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4whdv\" (UniqueName: \"kubernetes.io/projected/74d405d8-b58c-4358-be4d-f5a8276edb48-kube-api-access-4whdv\") pod \"maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.460948 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.460924 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:33.466830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.466800 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d61953-e352-44cd-9276-8285a2e665dc" path="/var/lib/kubelet/pods/91d61953-e352-44cd-9276-8285a2e665dc/volumes" Oct 30 22:18:33.467388 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.467367 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c201be91-d35d-4319-9fea-01c061905c27" path="/var/lib/kubelet/pods/c201be91-d35d-4319-9fea-01c061905c27/volumes" Oct 30 22:18:33.846414 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:18:33.846381 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d405d8_b58c_4358_be4d_f5a8276edb48.slice/crio-ec1eae6c4efe9b26d591e0d204e698ecf101d1d698d5eaa5ede5f19a50e892b1 WatchSource:0}: Error finding container ec1eae6c4efe9b26d591e0d204e698ecf101d1d698d5eaa5ede5f19a50e892b1: Status 404 returned error can't find the container with id ec1eae6c4efe9b26d591e0d204e698ecf101d1d698d5eaa5ede5f19a50e892b1 Oct 30 22:18:33.847216 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:33.847167 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf"] Oct 30 22:18:34.182360 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.182322 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" event={"ID":"74d405d8-b58c-4358-be4d-f5a8276edb48","Type":"ContainerStarted","Data":"80897248ce645d5c3353edfb11dfb54abd49464f58af8bbc0b26e8fd0b94a93a"} Oct 30 22:18:34.182481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.182420 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" event={"ID":"74d405d8-b58c-4358-be4d-f5a8276edb48","Type":"ContainerStarted","Data":"ec1eae6c4efe9b26d591e0d204e698ecf101d1d698d5eaa5ede5f19a50e892b1"} Oct 30 22:18:34.302767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.302710 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" podStartSLOduration=1.302697083 podStartE2EDuration="1.302697083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:34.300300436 +0000 UTC m=+2162.981450328" watchObservedRunningTime="2026-10-30 22:18:34.302697083 +0000 UTC m=+2162.983847002" Oct 30 22:18:34.420359 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.420269 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv"] Oct 30 22:18:34.424121 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.424106 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.481148 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.480623 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5"] Oct 30 22:18:34.489534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.485160 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv"] Oct 30 22:18:34.489534 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.485315 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.539006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.538971 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.541261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.539687 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbkxb\" (UniqueName: \"kubernetes.io/projected/1e7b3edf-3e86-427f-8905-62cd07310921-kube-api-access-pbkxb\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.541261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.539884 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.541261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.539994 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.541261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.540024 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.541261 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.540052 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5vck\" (UniqueName: \"kubernetes.io/projected/ef45a6fb-f758-4028-be3a-5813801497eb-kube-api-access-g5vck\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.543097 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.543059 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5"] Oct 30 22:18:34.641331 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.641301 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.641700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.641362 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.641700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.641393 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.641700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.641422 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5vck\" (UniqueName: \"kubernetes.io/projected/ef45a6fb-f758-4028-be3a-5813801497eb-kube-api-access-g5vck\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.641700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.641471 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.641700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.641556 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbkxb\" (UniqueName: \"kubernetes.io/projected/1e7b3edf-3e86-427f-8905-62cd07310921-kube-api-access-pbkxb\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.642680 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.642656 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.642796 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.642747 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.644303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.644286 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.644790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.644774 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.704633 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.704582 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbkxb\" (UniqueName: \"kubernetes.io/projected/1e7b3edf-3e86-427f-8905-62cd07310921-kube-api-access-pbkxb\") pod \"maas-api-revoke-sub-3b1683b9-mrcnv\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.705830 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.705808 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5vck\" (UniqueName: \"kubernetes.io/projected/ef45a6fb-f758-4028-be3a-5813801497eb-kube-api-access-g5vck\") pod \"maas-api-revoke-sub-e2418f41-cp7l5\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.714873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.714849 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs"] Oct 30 22:18:34.720150 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.720130 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.738848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.738823 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:34.776588 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.776558 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs"] Oct 30 22:18:34.801961 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.801933 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:34.850949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.845906 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvb6z\" (UniqueName: \"kubernetes.io/projected/e83e2764-b720-486e-a40d-4cff6193f5df-kube-api-access-nvb6z\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.850949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.845974 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.850949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.846067 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.947265 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.947229 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.947445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.947308 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvb6z\" (UniqueName: \"kubernetes.io/projected/e83e2764-b720-486e-a40d-4cff6193f5df-kube-api-access-nvb6z\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.947445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.947344 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.948079 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.948052 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:34.949780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:34.949757 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:35.006137 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.006082 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvb6z\" (UniqueName: \"kubernetes.io/projected/e83e2764-b720-486e-a40d-4cff6193f5df-kube-api-access-nvb6z\") pod \"maas-api-revoke-sub-22c0b420-6jsxs\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:35.036269 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.036242 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:35.139683 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.139648 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv"] Oct 30 22:18:35.178335 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:18:35.178302 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7b3edf_3e86_427f_8905_62cd07310921.slice/crio-4095be512433af570703ea4e913e28896afe65723bb0dee613251eebe3c23de8 WatchSource:0}: Error finding container 4095be512433af570703ea4e913e28896afe65723bb0dee613251eebe3c23de8: Status 404 returned error can't find the container with id 4095be512433af570703ea4e913e28896afe65723bb0dee613251eebe3c23de8 Oct 30 22:18:35.191001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.190968 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5"] Oct 30 22:18:35.191270 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:18:35.191185 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef45a6fb_f758_4028_be3a_5813801497eb.slice/crio-42a6be0934dc06cc40768e8d6644be79bde6ec0fd75a66edea1c5b869477224f WatchSource:0}: Error finding container 42a6be0934dc06cc40768e8d6644be79bde6ec0fd75a66edea1c5b869477224f: Status 404 returned error can't find the container with id 42a6be0934dc06cc40768e8d6644be79bde6ec0fd75a66edea1c5b869477224f Oct 30 22:18:35.191992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.191971 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" event={"ID":"1e7b3edf-3e86-427f-8905-62cd07310921","Type":"ContainerStarted","Data":"4095be512433af570703ea4e913e28896afe65723bb0dee613251eebe3c23de8"} Oct 30 22:18:35.193876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.193851 4933 generic.go:356] "Generic (PLEG): container finished" podID="74d405d8-b58c-4358-be4d-f5a8276edb48" containerID="80897248ce645d5c3353edfb11dfb54abd49464f58af8bbc0b26e8fd0b94a93a" exitCode=0 Oct 30 22:18:35.193964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.193906 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" event={"ID":"74d405d8-b58c-4358-be4d-f5a8276edb48","Type":"ContainerDied","Data":"80897248ce645d5c3353edfb11dfb54abd49464f58af8bbc0b26e8fd0b94a93a"} Oct 30 22:18:35.416287 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:18:35.416254 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83e2764_b720_486e_a40d_4cff6193f5df.slice/crio-3f4da2e041587fd39fb534eee6deada6811852e3c994e3fac7752e9390a6f529 WatchSource:0}: Error finding container 3f4da2e041587fd39fb534eee6deada6811852e3c994e3fac7752e9390a6f529: Status 404 returned error can't find the container with id 3f4da2e041587fd39fb534eee6deada6811852e3c994e3fac7752e9390a6f529 Oct 30 22:18:35.416915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:35.416881 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs"] Oct 30 22:18:36.202772 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.202746 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" event={"ID":"e83e2764-b720-486e-a40d-4cff6193f5df","Type":"ContainerStarted","Data":"6754d2b93ad0714cb9890c68be9f8f2deab7eea5f9409af254f6485bbe735493"} Oct 30 22:18:36.203004 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.202797 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" event={"ID":"e83e2764-b720-486e-a40d-4cff6193f5df","Type":"ContainerStarted","Data":"3f4da2e041587fd39fb534eee6deada6811852e3c994e3fac7752e9390a6f529"} Oct 30 22:18:36.204357 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.204326 4933 generic.go:356] "Generic (PLEG): container finished" podID="1e7b3edf-3e86-427f-8905-62cd07310921" containerID="88d37aa9426fdfd071dd0883cdb00b4b74716666dcffe8ccb194b8513009d899" exitCode=0 Oct 30 22:18:36.204443 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.204400 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" event={"ID":"1e7b3edf-3e86-427f-8905-62cd07310921","Type":"ContainerDied","Data":"88d37aa9426fdfd071dd0883cdb00b4b74716666dcffe8ccb194b8513009d899"} Oct 30 22:18:36.206054 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.206035 4933 generic.go:356] "Generic (PLEG): container finished" podID="ef45a6fb-f758-4028-be3a-5813801497eb" containerID="2567fc5f60ba9b3f3076865eb2c63c591277722e0362c72e04d79d1ee3e18f8c" exitCode=0 Oct 30 22:18:36.206160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.206102 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" event={"ID":"ef45a6fb-f758-4028-be3a-5813801497eb","Type":"ContainerDied","Data":"2567fc5f60ba9b3f3076865eb2c63c591277722e0362c72e04d79d1ee3e18f8c"} Oct 30 22:18:36.206160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.206135 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" event={"ID":"ef45a6fb-f758-4028-be3a-5813801497eb","Type":"ContainerStarted","Data":"42a6be0934dc06cc40768e8d6644be79bde6ec0fd75a66edea1c5b869477224f"} Oct 30 22:18:36.322149 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.322101 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" podStartSLOduration=2.322086995 podStartE2EDuration="2.322086995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.321732722 +0000 UTC m=+2165.002882615" watchObservedRunningTime="2026-10-30 22:18:36.322086995 +0000 UTC m=+2165.003236888" Oct 30 22:18:36.372806 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.372779 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:36.465755 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.465726 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-openshift-ca\") pod \"74d405d8-b58c-4358-be4d-f5a8276edb48\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " Oct 30 22:18:36.466090 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.465845 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-xks-ca\") pod \"74d405d8-b58c-4358-be4d-f5a8276edb48\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " Oct 30 22:18:36.466090 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.465967 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d405d8-b58c-4358-be4d-f5a8276edb48-kube-api-access-4whdv\" (UniqueName: \"kubernetes.io/projected/74d405d8-b58c-4358-be4d-f5a8276edb48-kube-api-access-4whdv\") pod \"74d405d8-b58c-4358-be4d-f5a8276edb48\" (UID: \"74d405d8-b58c-4358-be4d-f5a8276edb48\") " Oct 30 22:18:36.466090 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.466076 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-openshift-ca" pod "74d405d8-b58c-4358-be4d-f5a8276edb48" (UID: "74d405d8-b58c-4358-be4d-f5a8276edb48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:36.466348 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.466326 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:36.468102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.468069 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-xks-ca" pod "74d405d8-b58c-4358-be4d-f5a8276edb48" (UID: "74d405d8-b58c-4358-be4d-f5a8276edb48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:36.468221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.468147 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d405d8-b58c-4358-be4d-f5a8276edb48-kube-api-access-4whdv" pod "74d405d8-b58c-4358-be4d-f5a8276edb48" (UID: "74d405d8-b58c-4358-be4d-f5a8276edb48"). InnerVolumeSpecName "kube-api-access-4whdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:36.568249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.568212 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d405d8-b58c-4358-be4d-f5a8276edb48-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:36.568411 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:36.568253 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4whdv\" (UniqueName: \"kubernetes.io/projected/74d405d8-b58c-4358-be4d-f5a8276edb48-kube-api-access-4whdv\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:37.215072 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.215032 4933 generic.go:356] "Generic (PLEG): container finished" podID="e83e2764-b720-486e-a40d-4cff6193f5df" containerID="6754d2b93ad0714cb9890c68be9f8f2deab7eea5f9409af254f6485bbe735493" exitCode=0 Oct 30 22:18:37.215263 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.215113 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" event={"ID":"e83e2764-b720-486e-a40d-4cff6193f5df","Type":"ContainerDied","Data":"6754d2b93ad0714cb9890c68be9f8f2deab7eea5f9409af254f6485bbe735493"} Oct 30 22:18:37.216693 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.216671 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" Oct 30 22:18:37.216819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.216693 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf" event={"ID":"74d405d8-b58c-4358-be4d-f5a8276edb48","Type":"ContainerDied","Data":"ec1eae6c4efe9b26d591e0d204e698ecf101d1d698d5eaa5ede5f19a50e892b1"} Oct 30 22:18:37.216819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.216734 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec1eae6c4efe9b26d591e0d204e698ecf101d1d698d5eaa5ede5f19a50e892b1" Oct 30 22:18:37.419263 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.419236 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:37.421603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.421582 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:37.487212 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487155 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-xks-ca\") pod \"ef45a6fb-f758-4028-be3a-5813801497eb\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " Oct 30 22:18:37.487612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487232 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef45a6fb-f758-4028-be3a-5813801497eb-kube-api-access-g5vck\" (UniqueName: \"kubernetes.io/projected/ef45a6fb-f758-4028-be3a-5813801497eb-kube-api-access-g5vck\") pod \"ef45a6fb-f758-4028-be3a-5813801497eb\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " Oct 30 22:18:37.487612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487258 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-openshift-ca\") pod \"1e7b3edf-3e86-427f-8905-62cd07310921\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " Oct 30 22:18:37.487612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487339 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7b3edf-3e86-427f-8905-62cd07310921-kube-api-access-pbkxb\" (UniqueName: \"kubernetes.io/projected/1e7b3edf-3e86-427f-8905-62cd07310921-kube-api-access-pbkxb\") pod \"1e7b3edf-3e86-427f-8905-62cd07310921\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " Oct 30 22:18:37.487612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487469 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-openshift-ca\") pod \"ef45a6fb-f758-4028-be3a-5813801497eb\" (UID: \"ef45a6fb-f758-4028-be3a-5813801497eb\") " Oct 30 22:18:37.487612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487531 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-xks-ca\") pod \"1e7b3edf-3e86-427f-8905-62cd07310921\" (UID: \"1e7b3edf-3e86-427f-8905-62cd07310921\") " Oct 30 22:18:37.487902 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487708 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-openshift-ca" pod "1e7b3edf-3e86-427f-8905-62cd07310921" (UID: "1e7b3edf-3e86-427f-8905-62cd07310921"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:37.487902 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487768 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-openshift-ca" pod "ef45a6fb-f758-4028-be3a-5813801497eb" (UID: "ef45a6fb-f758-4028-be3a-5813801497eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:37.488023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487944 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:37.488023 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.487966 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:37.490096 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.490062 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-xks-ca" pod "ef45a6fb-f758-4028-be3a-5813801497eb" (UID: "ef45a6fb-f758-4028-be3a-5813801497eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:37.490220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.490150 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef45a6fb-f758-4028-be3a-5813801497eb-kube-api-access-g5vck" pod "ef45a6fb-f758-4028-be3a-5813801497eb" (UID: "ef45a6fb-f758-4028-be3a-5813801497eb"). InnerVolumeSpecName "kube-api-access-g5vck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:37.490220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.490179 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7b3edf-3e86-427f-8905-62cd07310921-kube-api-access-pbkxb" pod "1e7b3edf-3e86-427f-8905-62cd07310921" (UID: "1e7b3edf-3e86-427f-8905-62cd07310921"). InnerVolumeSpecName "kube-api-access-pbkxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:37.490399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.490382 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-xks-ca" pod "1e7b3edf-3e86-427f-8905-62cd07310921" (UID: "1e7b3edf-3e86-427f-8905-62cd07310921"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:37.589311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.589223 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbkxb\" (UniqueName: \"kubernetes.io/projected/1e7b3edf-3e86-427f-8905-62cd07310921-kube-api-access-pbkxb\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:37.589311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.589258 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7b3edf-3e86-427f-8905-62cd07310921-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:37.589311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.589272 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef45a6fb-f758-4028-be3a-5813801497eb-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:37.589311 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.589286 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5vck\" (UniqueName: \"kubernetes.io/projected/ef45a6fb-f758-4028-be3a-5813801497eb-kube-api-access-g5vck\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:37.802955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:37.802923 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:18:38.225798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.225723 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" Oct 30 22:18:38.225798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.225735 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv" event={"ID":"1e7b3edf-3e86-427f-8905-62cd07310921","Type":"ContainerDied","Data":"4095be512433af570703ea4e913e28896afe65723bb0dee613251eebe3c23de8"} Oct 30 22:18:38.225798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.225785 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4095be512433af570703ea4e913e28896afe65723bb0dee613251eebe3c23de8" Oct 30 22:18:38.227219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.227190 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" Oct 30 22:18:38.227345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.227237 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5" event={"ID":"ef45a6fb-f758-4028-be3a-5813801497eb","Type":"ContainerDied","Data":"42a6be0934dc06cc40768e8d6644be79bde6ec0fd75a66edea1c5b869477224f"} Oct 30 22:18:38.227345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.227286 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42a6be0934dc06cc40768e8d6644be79bde6ec0fd75a66edea1c5b869477224f" Oct 30 22:18:38.359087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.359058 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:38.497848 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.497787 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-xks-ca\") pod \"e83e2764-b720-486e-a40d-4cff6193f5df\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " Oct 30 22:18:38.498247 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.497961 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83e2764-b720-486e-a40d-4cff6193f5df-kube-api-access-nvb6z\" (UniqueName: \"kubernetes.io/projected/e83e2764-b720-486e-a40d-4cff6193f5df-kube-api-access-nvb6z\") pod \"e83e2764-b720-486e-a40d-4cff6193f5df\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " Oct 30 22:18:38.498247 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.498018 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-openshift-ca\") pod \"e83e2764-b720-486e-a40d-4cff6193f5df\" (UID: \"e83e2764-b720-486e-a40d-4cff6193f5df\") " Oct 30 22:18:38.498463 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.498441 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-openshift-ca" pod "e83e2764-b720-486e-a40d-4cff6193f5df" (UID: "e83e2764-b720-486e-a40d-4cff6193f5df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:38.500203 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.500175 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-xks-ca" pod "e83e2764-b720-486e-a40d-4cff6193f5df" (UID: "e83e2764-b720-486e-a40d-4cff6193f5df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:38.500414 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.500392 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83e2764-b720-486e-a40d-4cff6193f5df-kube-api-access-nvb6z" pod "e83e2764-b720-486e-a40d-4cff6193f5df" (UID: "e83e2764-b720-486e-a40d-4cff6193f5df"). InnerVolumeSpecName "kube-api-access-nvb6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:38.599592 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.599564 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvb6z\" (UniqueName: \"kubernetes.io/projected/e83e2764-b720-486e-a40d-4cff6193f5df-kube-api-access-nvb6z\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:38.599726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.599595 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:38.599726 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:38.599612 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83e2764-b720-486e-a40d-4cff6193f5df-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:39.236416 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:39.236381 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" Oct 30 22:18:39.236416 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:39.236406 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs" event={"ID":"e83e2764-b720-486e-a40d-4cff6193f5df","Type":"ContainerDied","Data":"3f4da2e041587fd39fb534eee6deada6811852e3c994e3fac7752e9390a6f529"} Oct 30 22:18:39.236699 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:39.236454 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f4da2e041587fd39fb534eee6deada6811852e3c994e3fac7752e9390a6f529" Oct 30 22:18:40.722455 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:40.722415 4933 scope.go:122] "RemoveContainer" containerID="e0f1a610e74bc288456b448553c1d759beeece942ffe7b2380b965bba263d622" Oct 30 22:18:41.091040 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:41.090978 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" podUID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:18:48.569631 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:48.569575 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 22:18:48.720083 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:48.720055 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-plugins-config-volume\") pod \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " Oct 30 22:18:48.720083 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:48.720088 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-kube-api-access-rs55h\" (UniqueName: \"kubernetes.io/projected/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-kube-api-access-rs55h\") pod \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\" (UID: \"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7\") " Oct 30 22:18:48.720552 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:48.720524 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-plugins-config-volume" pod "a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" (UID: "a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:48.720668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:48.720637 4933 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-plugins-config-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:48.722443 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:48.722420 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-kube-api-access-rs55h" pod "a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" (UID: "a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7"). InnerVolumeSpecName "kube-api-access-rs55h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:48.821638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:48.821573 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs55h\" (UniqueName: \"kubernetes.io/projected/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7-kube-api-access-rs55h\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:18:49.331998 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.331960 4933 generic.go:356] "Generic (PLEG): container finished" podID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" containerID="08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8" exitCode=0 Oct 30 22:18:49.332179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.332009 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" Oct 30 22:18:49.332179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.332048 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" event={"ID":"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7","Type":"ContainerDied","Data":"08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8"} Oct 30 22:18:49.332179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.332115 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm" event={"ID":"a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7","Type":"ContainerDied","Data":"9dfc3b362c6a450bfabea2ad6e413c198a35e43fc25d85cd75ae06bea8459a1d"} Oct 30 22:18:49.332179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.332145 4933 scope.go:122] "RemoveContainer" containerID="08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8" Oct 30 22:18:49.345357 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.345335 4933 scope.go:122] "RemoveContainer" containerID="08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8" Oct 30 22:18:49.345663 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:49.345643 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8\": container with ID starting with 08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8 not found: ID does not exist" containerID="08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8" Oct 30 22:18:49.345761 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.345671 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8"} err="failed to get container status \"08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8\": rpc error: code = NotFound desc = could not find container \"08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8\": container with ID starting with 08d9347a888c5102dcbba61eb43a5e71f25063856b9f93bc162ea8ef5e7282b8 not found: ID does not exist" Oct 30 22:18:49.499218 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.499188 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm"] Oct 30 22:18:49.550401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:49.550365 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9f8aea-fdfdfdbbf-cm9rm"] Oct 30 22:18:51.466614 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:51.466577 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" path="/var/lib/kubelet/pods/a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7/volumes" Oct 30 22:18:52.053068 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:52.053028 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb"] Oct 30 22:18:52.100587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:52.100554 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4864ff62-dd7nb"] Oct 30 22:18:53.465729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:53.465694 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a" path="/var/lib/kubelet/pods/a0c1b1d1-374c-425e-b7dc-c8d95df3ed0a/volumes" Oct 30 22:18:56.932805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:56.932762 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl"] Oct 30 22:18:56.980950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:56.980919 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dde1e8-9v8pl"] Oct 30 22:18:57.464970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:57.464943 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1740003e-b969-4634-90e0-1ef546ce7d4e" path="/var/lib/kubelet/pods/1740003e-b969-4634-90e0-1ef546ce7d4e/volumes" Oct 30 22:18:58.280481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:58.280434 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c"] Oct 30 22:18:58.332177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:58.332148 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9f8aea-lq27c"] Oct 30 22:18:58.547852 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:18:58.547771 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:18:59.465974 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:18:59.465943 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23284b96-dc1f-4cd1-a1c9-affdc073a307" path="/var/lib/kubelet/pods/23284b96-dc1f-4cd1-a1c9-affdc073a307/volumes" Oct 30 22:19:04.884022 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:04.883989 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf"] Oct 30 22:19:04.934301 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:04.934273 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-242ddaf0-v42gf"] Oct 30 22:19:05.324423 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:05.324386 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9"] Oct 30 22:19:05.324648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:05.324601 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" containerID="cri-o://7ad304d198365df8584cdc82fda141af4d0e4a992a6f480a17e626ce7bca1c36" gracePeriod=30 Oct 30 22:19:05.471341 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:05.471310 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d405d8-b58c-4358-be4d-f5a8276edb48" path="/var/lib/kubelet/pods/74d405d8-b58c-4358-be4d-f5a8276edb48/volumes" Oct 30 22:19:10.510299 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.510264 4933 generic.go:356] "Generic (PLEG): container finished" podID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerID="7ad304d198365df8584cdc82fda141af4d0e4a992a6f480a17e626ce7bca1c36" exitCode=0 Oct 30 22:19:10.510712 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.510339 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" event={"ID":"e090da55-55cc-45c6-bf58-8c5f17818d13","Type":"ContainerDied","Data":"7ad304d198365df8584cdc82fda141af4d0e4a992a6f480a17e626ce7bca1c36"} Oct 30 22:19:10.595935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.595883 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 22:19:10.684231 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684206 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-kube-api-access-6sw6d\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-kube-api-access-6sw6d\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684248 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-credential-socket\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684289 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-data\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684458 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684392 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-envoy\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684476 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-podinfo\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684604 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684571 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-certs\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684608 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-credential-socket" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.684670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684620 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e090da55-55cc-45c6-bf58-8c5f17818d13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e090da55-55cc-45c6-bf58-8c5f17818d13-istiod-ca-cert\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684670 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684649 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-data" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.684841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684691 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-token\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-token\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684731 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-socket\") pod \"e090da55-55cc-45c6-bf58-8c5f17818d13\" (UID: \"e090da55-55cc-45c6-bf58-8c5f17818d13\") " Oct 30 22:19:10.684841 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684824 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-certs" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.685015 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.684926 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e090da55-55cc-45c6-bf58-8c5f17818d13-istiod-ca-cert" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:10.685192 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.685030 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-socket" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.685192 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.685104 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.685370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.685200 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.685370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.685214 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.686569 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.686549 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-kube-api-access-6sw6d" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "kube-api-access-6sw6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:10.686666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.686651 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-envoy" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:10.687238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.687211 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-podinfo" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:10.687238 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.687221 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-token" pod "e090da55-55cc-45c6-bf58-8c5f17818d13" (UID: "e090da55-55cc-45c6-bf58-8c5f17818d13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:10.785983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.785947 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.785983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.785972 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.785983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.785981 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e090da55-55cc-45c6-bf58-8c5f17818d13-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.785983 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.785989 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.786319 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.785999 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e090da55-55cc-45c6-bf58-8c5f17818d13-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:10.786319 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:10.786007 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sw6d\" (UniqueName: \"kubernetes.io/projected/e090da55-55cc-45c6-bf58-8c5f17818d13-kube-api-access-6sw6d\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:11.517931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:11.517903 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" Oct 30 22:19:11.518310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:11.517959 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9" event={"ID":"e090da55-55cc-45c6-bf58-8c5f17818d13","Type":"ContainerDied","Data":"39353c61dacb3b0378ed81401db7f36da1caf8ed5bffb5f5aed92a96c70ac25d"} Oct 30 22:19:11.518310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:11.518013 4933 scope.go:122] "RemoveContainer" containerID="7ad304d198365df8584cdc82fda141af4d0e4a992a6f480a17e626ce7bca1c36" Oct 30 22:19:11.737762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:11.737694 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9"] Oct 30 22:19:11.742788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:11.742767 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9f8aea-openshift-default-8fc8fc847-nz5k9"] Oct 30 22:19:13.468056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:13.468017 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" path="/var/lib/kubelet/pods/e090da55-55cc-45c6-bf58-8c5f17818d13/volumes" Oct 30 22:19:17.067700 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:17.067662 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:19:28.545712 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:19:28.545670 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:19:46.555065 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555027 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z"] Oct 30 22:19:46.555538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555470 4933 cpu_manager.go:414] "containerMap: removing container" podUID="74d405d8-b58c-4358-be4d-f5a8276edb48" containerName="revoke-keys" Oct 30 22:19:46.555538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555480 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d405d8-b58c-4358-be4d-f5a8276edb48" containerName="revoke-keys" Oct 30 22:19:46.555538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555517 4933 cpu_manager.go:414] "containerMap: removing container" podUID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" containerName="payload-processing" Oct 30 22:19:46.555538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555525 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" containerName="payload-processing" Oct 30 22:19:46.555538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555531 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e83e2764-b720-486e-a40d-4cff6193f5df" containerName="revoke-keys" Oct 30 22:19:46.555538 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555535 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83e2764-b720-486e-a40d-4cff6193f5df" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555550 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ef45a6fb-f758-4028-be3a-5813801497eb" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555554 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef45a6fb-f758-4028-be3a-5813801497eb" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555564 4933 cpu_manager.go:414] "containerMap: removing container" podUID="1e7b3edf-3e86-427f-8905-62cd07310921" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555569 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7b3edf-3e86-427f-8905-62cd07310921" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555578 4933 cpu_manager.go:414] "containerMap: removing container" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555582 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555649 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d405d8-b58c-4358-be4d-f5a8276edb48" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555658 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c28fcf-5e62-4b4d-ae03-c7e16b464bc7" containerName="payload-processing" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555665 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83e2764-b720-486e-a40d-4cff6193f5df" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555672 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef45a6fb-f758-4028-be3a-5813801497eb" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555678 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7b3edf-3e86-427f-8905-62cd07310921" containerName="revoke-keys" Oct 30 22:19:46.555767 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.555683 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="e090da55-55cc-45c6-bf58-8c5f17818d13" containerName="istio-proxy" Oct 30 22:19:46.560219 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.560204 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.619978 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.619955 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.620094 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.620017 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhngs\" (UniqueName: \"kubernetes.io/projected/3e32866a-adfb-49c8-808b-a71c9921d1d9-kube-api-access-zhngs\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.620168 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.620147 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.645296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.645265 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z"] Oct 30 22:19:46.689303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.689273 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:46.689473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.689379 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:19:46.689473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.689451 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:19:46.689790 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.689777 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:19:46.720851 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.720828 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.720957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.720886 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhngs\" (UniqueName: \"kubernetes.io/projected/3e32866a-adfb-49c8-808b-a71c9921d1d9-kube-api-access-zhngs\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.720957 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.720946 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.721649 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.721632 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.723399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.723371 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.807722 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.807667 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhngs\" (UniqueName: \"kubernetes.io/projected/3e32866a-adfb-49c8-808b-a71c9921d1d9-kube-api-access-zhngs\") pod \"maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:46.872533 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:46.872485 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:47.245978 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:47.245955 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z"] Oct 30 22:19:47.247125 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:19:47.247094 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e32866a_adfb_49c8_808b_a71c9921d1d9.slice/crio-b04c857826c0d7a3285a5e9d11d09b53cfe8f580ccf2bf039b5399ff02be6271 WatchSource:0}: Error finding container b04c857826c0d7a3285a5e9d11d09b53cfe8f580ccf2bf039b5399ff02be6271: Status 404 returned error can't find the container with id b04c857826c0d7a3285a5e9d11d09b53cfe8f580ccf2bf039b5399ff02be6271 Oct 30 22:19:47.775668 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:47.775631 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" event={"ID":"3e32866a-adfb-49c8-808b-a71c9921d1d9","Type":"ContainerStarted","Data":"3d605b3c0c32fc7b09afed9cdeceec44fe6aa99c29c33cf588c9f1493fe6b040"} Oct 30 22:19:47.776185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:47.775715 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" event={"ID":"3e32866a-adfb-49c8-808b-a71c9921d1d9","Type":"ContainerStarted","Data":"b04c857826c0d7a3285a5e9d11d09b53cfe8f580ccf2bf039b5399ff02be6271"} Oct 30 22:19:47.908364 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:47.908259 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" podStartSLOduration=1.9082444029999999 podStartE2EDuration="1.908244403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:47.907247428 +0000 UTC m=+2236.588397347" watchObservedRunningTime="2026-10-30 22:19:47.908244403 +0000 UTC m=+2236.589394296" Oct 30 22:19:48.784041 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:48.784010 4933 generic.go:356] "Generic (PLEG): container finished" podID="3e32866a-adfb-49c8-808b-a71c9921d1d9" containerID="3d605b3c0c32fc7b09afed9cdeceec44fe6aa99c29c33cf588c9f1493fe6b040" exitCode=0 Oct 30 22:19:48.784416 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:48.784087 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" event={"ID":"3e32866a-adfb-49c8-808b-a71c9921d1d9","Type":"ContainerDied","Data":"3d605b3c0c32fc7b09afed9cdeceec44fe6aa99c29c33cf588c9f1493fe6b040"} Oct 30 22:19:49.951600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:49.951570 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:50.058611 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.058545 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-openshift-ca\") pod \"3e32866a-adfb-49c8-808b-a71c9921d1d9\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " Oct 30 22:19:50.058742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.058653 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-xks-ca\") pod \"3e32866a-adfb-49c8-808b-a71c9921d1d9\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " Oct 30 22:19:50.058742 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.058701 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e32866a-adfb-49c8-808b-a71c9921d1d9-kube-api-access-zhngs\" (UniqueName: \"kubernetes.io/projected/3e32866a-adfb-49c8-808b-a71c9921d1d9-kube-api-access-zhngs\") pod \"3e32866a-adfb-49c8-808b-a71c9921d1d9\" (UID: \"3e32866a-adfb-49c8-808b-a71c9921d1d9\") " Oct 30 22:19:50.058934 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.058910 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-openshift-ca" pod "3e32866a-adfb-49c8-808b-a71c9921d1d9" (UID: "3e32866a-adfb-49c8-808b-a71c9921d1d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:50.059138 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.059120 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:50.060931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.060902 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-xks-ca" pod "3e32866a-adfb-49c8-808b-a71c9921d1d9" (UID: "3e32866a-adfb-49c8-808b-a71c9921d1d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:50.061031 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.060931 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e32866a-adfb-49c8-808b-a71c9921d1d9-kube-api-access-zhngs" pod "3e32866a-adfb-49c8-808b-a71c9921d1d9" (UID: "3e32866a-adfb-49c8-808b-a71c9921d1d9"). InnerVolumeSpecName "kube-api-access-zhngs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:50.160386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.160357 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e32866a-adfb-49c8-808b-a71c9921d1d9-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:50.160386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.160387 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhngs\" (UniqueName: \"kubernetes.io/projected/3e32866a-adfb-49c8-808b-a71c9921d1d9-kube-api-access-zhngs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:19:50.802425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.802395 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" Oct 30 22:19:50.802627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.802439 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z" event={"ID":"3e32866a-adfb-49c8-808b-a71c9921d1d9","Type":"ContainerDied","Data":"b04c857826c0d7a3285a5e9d11d09b53cfe8f580ccf2bf039b5399ff02be6271"} Oct 30 22:19:50.802627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:50.802484 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04c857826c0d7a3285a5e9d11d09b53cfe8f580ccf2bf039b5399ff02be6271" Oct 30 22:19:51.031603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:51.031578 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp"] Oct 30 22:19:58.129034 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.129001 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm"] Oct 30 22:19:58.543108 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:19:58.543075 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:19:58.619484 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.619448 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8"] Oct 30 22:19:58.619985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.619967 4933 cpu_manager.go:414] "containerMap: removing container" podUID="3e32866a-adfb-49c8-808b-a71c9921d1d9" containerName="revoke-keys" Oct 30 22:19:58.619985 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.619985 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e32866a-adfb-49c8-808b-a71c9921d1d9" containerName="revoke-keys" Oct 30 22:19:58.620122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.620065 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e32866a-adfb-49c8-808b-a71c9921d1d9" containerName="revoke-keys" Oct 30 22:19:58.623343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.623321 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.679446 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.679425 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:19:58.679627 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.679608 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:19:58.680044 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.680021 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:19:58.680221 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.680131 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:19:58.683431 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.683410 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8"] Oct 30 22:19:58.745931 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.745904 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sxf\" (UniqueName: \"kubernetes.io/projected/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-kube-api-access-k8sxf\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.746078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.745939 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.746187 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.746165 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.847586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.847520 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.847709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.847622 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sxf\" (UniqueName: \"kubernetes.io/projected/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-kube-api-access-k8sxf\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.847709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.847654 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.848366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.848328 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.850365 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.850343 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.905658 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.905633 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sxf\" (UniqueName: \"kubernetes.io/projected/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-kube-api-access-k8sxf\") pod \"maas-api-revoke-sub-dbb9f33d-rqfz8\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:58.936622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:58.936597 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:19:59.311844 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:19:59.311805 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937b21e4_e2db_4d0b_bd1a_8af39cc2b564.slice/crio-e33be1e3a0fe7f7cb5f9aa38d90e924de9d5695fead1a69cf2785cbfae27eabc WatchSource:0}: Error finding container e33be1e3a0fe7f7cb5f9aa38d90e924de9d5695fead1a69cf2785cbfae27eabc: Status 404 returned error can't find the container with id e33be1e3a0fe7f7cb5f9aa38d90e924de9d5695fead1a69cf2785cbfae27eabc Oct 30 22:19:59.312175 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:59.311987 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8"] Oct 30 22:19:59.866560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:59.866517 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" event={"ID":"937b21e4-e2db-4d0b-bd1a-8af39cc2b564","Type":"ContainerStarted","Data":"d78243ab5b46564bd4b099d91798a10bbab990f031bebf0837adb86d2e9ff7df"} Oct 30 22:19:59.866757 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:59.866613 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" event={"ID":"937b21e4-e2db-4d0b-bd1a-8af39cc2b564","Type":"ContainerStarted","Data":"e33be1e3a0fe7f7cb5f9aa38d90e924de9d5695fead1a69cf2785cbfae27eabc"} Oct 30 22:19:59.983547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:19:59.983469 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" podStartSLOduration=1.983455487 podStartE2EDuration="1.983455487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:59.982606251 +0000 UTC m=+2248.663756146" watchObservedRunningTime="2026-10-30 22:19:59.983455487 +0000 UTC m=+2248.664605379" Oct 30 22:20:00.165306 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:00.165257 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5"] Oct 30 22:20:00.165578 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:00.165535 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" containerID="cri-o://a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a" gracePeriod=30 Oct 30 22:20:00.875240 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:00.875205 4933 generic.go:356] "Generic (PLEG): container finished" podID="937b21e4-e2db-4d0b-bd1a-8af39cc2b564" containerID="d78243ab5b46564bd4b099d91798a10bbab990f031bebf0837adb86d2e9ff7df" exitCode=0 Oct 30 22:20:00.875638 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:00.875282 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" event={"ID":"937b21e4-e2db-4d0b-bd1a-8af39cc2b564","Type":"ContainerDied","Data":"d78243ab5b46564bd4b099d91798a10bbab990f031bebf0837adb86d2e9ff7df"} Oct 30 22:20:01.674444 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:01.674410 4933 patch_prober.go:29] interesting pod/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:20:01.674656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:01.674462 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:20:02.018591 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.018567 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:20:02.179155 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.179129 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-openshift-ca\") pod \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " Oct 30 22:20:02.179324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.179236 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-kube-api-access-k8sxf\" (UniqueName: \"kubernetes.io/projected/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-kube-api-access-k8sxf\") pod \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " Oct 30 22:20:02.179324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.179316 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-xks-ca\") pod \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\" (UID: \"937b21e4-e2db-4d0b-bd1a-8af39cc2b564\") " Oct 30 22:20:02.179583 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.179560 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-openshift-ca" pod "937b21e4-e2db-4d0b-bd1a-8af39cc2b564" (UID: "937b21e4-e2db-4d0b-bd1a-8af39cc2b564"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:02.181503 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.181459 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-xks-ca" pod "937b21e4-e2db-4d0b-bd1a-8af39cc2b564" (UID: "937b21e4-e2db-4d0b-bd1a-8af39cc2b564"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:02.181621 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.181576 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-kube-api-access-k8sxf" pod "937b21e4-e2db-4d0b-bd1a-8af39cc2b564" (UID: "937b21e4-e2db-4d0b-bd1a-8af39cc2b564"). InnerVolumeSpecName "kube-api-access-k8sxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:02.281025 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.280995 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:02.281025 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.281021 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:02.281025 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.281032 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sxf\" (UniqueName: \"kubernetes.io/projected/937b21e4-e2db-4d0b-bd1a-8af39cc2b564-kube-api-access-k8sxf\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:02.890426 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.890398 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" Oct 30 22:20:02.890609 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.890438 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8" event={"ID":"937b21e4-e2db-4d0b-bd1a-8af39cc2b564","Type":"ContainerDied","Data":"e33be1e3a0fe7f7cb5f9aa38d90e924de9d5695fead1a69cf2785cbfae27eabc"} Oct 30 22:20:02.890609 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:02.890484 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33be1e3a0fe7f7cb5f9aa38d90e924de9d5695fead1a69cf2785cbfae27eabc" Oct 30 22:20:03.130108 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:03.130069 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" podUID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" containerName="payload-pre-processing" containerID="cri-o://7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f" gracePeriod=25 Oct 30 22:20:04.108154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:04.108119 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" podUID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:20:05.435417 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.435364 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:20:05.510471 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510442 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386b34f2-8410-4621-9678-4c7a1df8270f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386b34f2-8410-4621-9678-4c7a1df8270f-istiod-ca-cert\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510608 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510511 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/386b34f2-8410-4621-9678-4c7a1df8270f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386b34f2-8410-4621-9678-4c7a1df8270f-istio-podinfo\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510608 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510576 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-socket\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510608 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510598 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-istio-token\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-istio-token\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510632 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-kube-api-access-jhv2c\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-kube-api-access-jhv2c\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510655 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-data\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510729 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510695 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-credential-socket\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510893 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510743 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-envoy\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.510893 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510760 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386b34f2-8410-4621-9678-4c7a1df8270f-istiod-ca-cert" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:05.510893 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510780 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-certs\") pod \"386b34f2-8410-4621-9678-4c7a1df8270f\" (UID: \"386b34f2-8410-4621-9678-4c7a1df8270f\") " Oct 30 22:20:05.511063 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.510939 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-socket" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:05.511257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.511229 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-credential-socket" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:05.511257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.511250 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386b34f2-8410-4621-9678-4c7a1df8270f-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.511257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.511257 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-data" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:05.511257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.511273 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.511743 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.511341 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-certs" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:05.513058 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.513033 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-istio-token" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:05.513183 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.513060 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-kube-api-access-jhv2c" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "kube-api-access-jhv2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:05.513183 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.513133 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/386b34f2-8410-4621-9678-4c7a1df8270f-istio-podinfo" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:05.513412 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.513397 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-envoy" pod "386b34f2-8410-4621-9678-4c7a1df8270f" (UID: "386b34f2-8410-4621-9678-4c7a1df8270f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:05.612248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.612224 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386b34f2-8410-4621-9678-4c7a1df8270f-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.612248 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.612249 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.612424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.612259 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhv2c\" (UniqueName: \"kubernetes.io/projected/386b34f2-8410-4621-9678-4c7a1df8270f-kube-api-access-jhv2c\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.612424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.612268 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.612424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.612277 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.612424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.612287 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.612424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.612296 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386b34f2-8410-4621-9678-4c7a1df8270f-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:05.916095 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.916057 4933 generic.go:356] "Generic (PLEG): container finished" podID="386b34f2-8410-4621-9678-4c7a1df8270f" containerID="a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a" exitCode=0 Oct 30 22:20:05.916296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.916128 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" Oct 30 22:20:05.916296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.916117 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" event={"ID":"386b34f2-8410-4621-9678-4c7a1df8270f","Type":"ContainerDied","Data":"a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a"} Oct 30 22:20:05.916296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.916199 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5" event={"ID":"386b34f2-8410-4621-9678-4c7a1df8270f","Type":"ContainerDied","Data":"4cfca5a87521ff64d9dc26b04eb6c8edb8ef4256500cdc7a1e9c85d6a033db5b"} Oct 30 22:20:05.916296 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.916228 4933 scope.go:122] "RemoveContainer" containerID="a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a" Oct 30 22:20:05.927999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.927981 4933 scope.go:122] "RemoveContainer" containerID="c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4" Oct 30 22:20:05.943454 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.943435 4933 scope.go:122] "RemoveContainer" containerID="a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a" Oct 30 22:20:05.943821 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:20:05.943798 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a\": container with ID starting with a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a not found: ID does not exist" containerID="a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a" Oct 30 22:20:05.943899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.943830 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a"} err="failed to get container status \"a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a\": rpc error: code = NotFound desc = could not find container \"a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a\": container with ID starting with a417df9689d9ca8d7e0079dd0898f7b70c9549f059ad63e8fd201f81efa0005a not found: ID does not exist" Oct 30 22:20:05.943899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.943850 4933 scope.go:122] "RemoveContainer" containerID="c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4" Oct 30 22:20:05.944129 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:20:05.944107 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4\": container with ID starting with c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4 not found: ID does not exist" containerID="c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4" Oct 30 22:20:05.944202 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:05.944137 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4"} err="failed to get container status \"c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4\": rpc error: code = NotFound desc = could not find container \"c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4\": container with ID starting with c1acce919a0574ca2781222caba051f682c72adc9f2d18c0fdc29e92c54d44c4 not found: ID does not exist" Oct 30 22:20:06.054033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.053990 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" containerID="cri-o://29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d" gracePeriod=45 Oct 30 22:20:06.084992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.084970 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5"] Oct 30 22:20:06.136654 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.136627 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-242ddaf0-openshift-default-79b7ddb6f7-frnd5"] Oct 30 22:20:06.155846 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.155811 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": dial tcp 10.132.0.58:8000: connect: connection refused" Oct 30 22:20:06.321153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.321117 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:20:06.419587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.419560 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-dshm\") pod \"ab0a49b7-dea2-4923-bd0a-30e48455c795\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " Oct 30 22:20:06.419764 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.419650 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab0a49b7-dea2-4923-bd0a-30e48455c795-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0a49b7-dea2-4923-bd0a-30e48455c795-tls-certs\") pod \"ab0a49b7-dea2-4923-bd0a-30e48455c795\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " Oct 30 22:20:06.419764 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.419690 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0a49b7-dea2-4923-bd0a-30e48455c795-kube-api-access-zcgs5\" (UniqueName: \"kubernetes.io/projected/ab0a49b7-dea2-4923-bd0a-30e48455c795-kube-api-access-zcgs5\") pod \"ab0a49b7-dea2-4923-bd0a-30e48455c795\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " Oct 30 22:20:06.419900 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.419858 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-model-cache\") pod \"ab0a49b7-dea2-4923-bd0a-30e48455c795\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " Oct 30 22:20:06.420001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.419977 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-home\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-home\") pod \"ab0a49b7-dea2-4923-bd0a-30e48455c795\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " Oct 30 22:20:06.420001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.420028 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-tmp-dir\") pod \"ab0a49b7-dea2-4923-bd0a-30e48455c795\" (UID: \"ab0a49b7-dea2-4923-bd0a-30e48455c795\") " Oct 30 22:20:06.420247 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.420045 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-model-cache" pod "ab0a49b7-dea2-4923-bd0a-30e48455c795" (UID: "ab0a49b7-dea2-4923-bd0a-30e48455c795"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.420447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.420429 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.420559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.420426 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-home" pod "ab0a49b7-dea2-4923-bd0a-30e48455c795" (UID: "ab0a49b7-dea2-4923-bd0a-30e48455c795"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.420559 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.420474 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-tmp-dir" pod "ab0a49b7-dea2-4923-bd0a-30e48455c795" (UID: "ab0a49b7-dea2-4923-bd0a-30e48455c795"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.422630 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.422575 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0a49b7-dea2-4923-bd0a-30e48455c795-kube-api-access-zcgs5" pod "ab0a49b7-dea2-4923-bd0a-30e48455c795" (UID: "ab0a49b7-dea2-4923-bd0a-30e48455c795"). InnerVolumeSpecName "kube-api-access-zcgs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:06.423287 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.423261 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-dshm" pod "ab0a49b7-dea2-4923-bd0a-30e48455c795" (UID: "ab0a49b7-dea2-4923-bd0a-30e48455c795"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:06.423616 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.423593 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0a49b7-dea2-4923-bd0a-30e48455c795-tls-certs" pod "ab0a49b7-dea2-4923-bd0a-30e48455c795" (UID: "ab0a49b7-dea2-4923-bd0a-30e48455c795"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:06.521393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.521361 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.521393 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.521389 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.521838 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.521402 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab0a49b7-dea2-4923-bd0a-30e48455c795-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.521838 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.521414 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab0a49b7-dea2-4923-bd0a-30e48455c795-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.521838 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.521429 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcgs5\" (UniqueName: \"kubernetes.io/projected/ab0a49b7-dea2-4923-bd0a-30e48455c795-kube-api-access-zcgs5\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:06.925730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.925693 4933 generic.go:356] "Generic (PLEG): container finished" podID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerID="29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d" exitCode=0 Oct 30 22:20:06.925907 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.925737 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" Oct 30 22:20:06.925907 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.925772 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" event={"ID":"ab0a49b7-dea2-4923-bd0a-30e48455c795","Type":"ContainerDied","Data":"29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d"} Oct 30 22:20:06.925907 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.925810 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp" event={"ID":"ab0a49b7-dea2-4923-bd0a-30e48455c795","Type":"ContainerDied","Data":"d73808f22e9c9ea61e855bace0ec1b6837d5bc7503bd9eac552159b5841324bc"} Oct 30 22:20:06.925907 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.925832 4933 scope.go:122] "RemoveContainer" containerID="29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d" Oct 30 22:20:06.937871 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.937850 4933 scope.go:122] "RemoveContainer" containerID="29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d" Oct 30 22:20:06.938204 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:20:06.938181 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d\": container with ID starting with 29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d not found: ID does not exist" containerID="29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d" Oct 30 22:20:06.938274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:06.938216 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d"} err="failed to get container status \"29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d\": rpc error: code = NotFound desc = could not find container \"29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d\": container with ID starting with 29dedffc3f627fe4d888bb17439826d9651e0d3253d686b78c85d650fc8f658d not found: ID does not exist" Oct 30 22:20:07.092995 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:07.092964 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp"] Oct 30 22:20:07.143664 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:07.143634 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-29cbad/simulator-w6-29cbad-kserve-55df6bd77d-rb7fp"] Oct 30 22:20:07.467064 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:07.467034 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" path="/var/lib/kubelet/pods/386b34f2-8410-4621-9678-4c7a1df8270f/volumes" Oct 30 22:20:07.467453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:07.467439 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" path="/var/lib/kubelet/pods/ab0a49b7-dea2-4923-bd0a-30e48455c795/volumes" Oct 30 22:20:14.108986 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:14.108948 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" podUID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:20:15.654687 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:15.654649 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z"] Oct 30 22:20:15.704190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:15.704149 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-29cbad-nbp7z"] Oct 30 22:20:17.466421 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:17.466389 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e32866a-adfb-49c8-808b-a71c9921d1d9" path="/var/lib/kubelet/pods/3e32866a-adfb-49c8-808b-a71c9921d1d9/volumes" Oct 30 22:20:23.353818 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.353781 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76"] Oct 30 22:20:23.354291 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.354097 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" containerID="cri-o://3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65" gracePeriod=30 Oct 30 22:20:23.390418 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.390356 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 22:20:23.484142 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.484109 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-plugins-config-volume\") pod \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " Oct 30 22:20:23.484332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.484202 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-kube-api-access-xw7j7\" (UniqueName: \"kubernetes.io/projected/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-kube-api-access-xw7j7\") pod \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\" (UID: \"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266\") " Oct 30 22:20:23.484594 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.484564 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-plugins-config-volume" pod "0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" (UID: "0e696cd7-f7b8-4da0-bf63-00b0ecd3c266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:23.486964 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.486932 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-kube-api-access-xw7j7" pod "0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" (UID: "0e696cd7-f7b8-4da0-bf63-00b0ecd3c266"). InnerVolumeSpecName "kube-api-access-xw7j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:23.585896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.585867 4933 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-plugins-config-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:23.585896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:23.585896 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7j7\" (UniqueName: \"kubernetes.io/projected/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266-kube-api-access-xw7j7\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:24.105836 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.105806 4933 generic.go:356] "Generic (PLEG): container finished" podID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" containerID="7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f" exitCode=0 Oct 30 22:20:24.106006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.105849 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" Oct 30 22:20:24.106006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.105890 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" event={"ID":"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266","Type":"ContainerDied","Data":"7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f"} Oct 30 22:20:24.106006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.105939 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm" event={"ID":"0e696cd7-f7b8-4da0-bf63-00b0ecd3c266","Type":"ContainerDied","Data":"45c9cdd2472aa234ea14219ea22a469c5d0eb806d6165e6c6442608940834617"} Oct 30 22:20:24.106006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.105967 4933 scope.go:122] "RemoveContainer" containerID="7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f" Oct 30 22:20:24.117207 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.117189 4933 scope.go:122] "RemoveContainer" containerID="7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f" Oct 30 22:20:24.117459 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:20:24.117443 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f\": container with ID starting with 7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f not found: ID does not exist" containerID="7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f" Oct 30 22:20:24.117548 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.117467 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f"} err="failed to get container status \"7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f\": rpc error: code = NotFound desc = could not find container \"7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f\": container with ID starting with 7a07708a3c88c3bae0464a2b06b8aeadf3301bb61b88092d2b4427f86e9cd58f not found: ID does not exist" Oct 30 22:20:24.287335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.287310 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm"] Oct 30 22:20:24.337935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:24.337901 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-29cbad-6447997cd7-h89bm"] Oct 30 22:20:25.465527 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:25.465474 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" path="/var/lib/kubelet/pods/0e696cd7-f7b8-4da0-bf63-00b0ecd3c266/volumes" Oct 30 22:20:28.555856 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:20:28.555804 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:20:28.604441 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.604418 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 22:20:28.737112 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737051 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-data\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737112 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737104 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-certs\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737135 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istiod-ca-cert\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737175 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-podinfo\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737223 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-socket\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737279 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-envoy\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737312 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-token\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737340 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-kube-api-access-5nlbh\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-kube-api-access-5nlbh\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737326 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-data" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.737679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737407 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-certs" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.737679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737413 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-credential-socket\") pod \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\" (UID: \"b2bf740a-f41e-46a3-8fcf-7b94cf50919f\") " Oct 30 22:20:28.737679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737560 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istiod-ca-cert" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:28.737918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737702 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-credential-socket" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.737918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737732 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-socket" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.737918 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737897 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.738077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.737918 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.738077 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.738010 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.738158 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.738108 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.738158 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.738122 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.739682 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.739650 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-envoy" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:28.739811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.739691 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-podinfo" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:28.739811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.739776 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-kube-api-access-5nlbh" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "kube-api-access-5nlbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:28.740157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.740138 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-token" pod "b2bf740a-f41e-46a3-8fcf-7b94cf50919f" (UID: "b2bf740a-f41e-46a3-8fcf-7b94cf50919f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:28.839270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.839245 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.839270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.839266 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.839270 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.839275 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:28.839549 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:28.839284 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nlbh\" (UniqueName: \"kubernetes.io/projected/b2bf740a-f41e-46a3-8fcf-7b94cf50919f-kube-api-access-5nlbh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:29.144970 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.144940 4933 generic.go:356] "Generic (PLEG): container finished" podID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerID="3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65" exitCode=0 Oct 30 22:20:29.145157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.144987 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" Oct 30 22:20:29.145157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.145024 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" event={"ID":"b2bf740a-f41e-46a3-8fcf-7b94cf50919f","Type":"ContainerDied","Data":"3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65"} Oct 30 22:20:29.145157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.145117 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76" event={"ID":"b2bf740a-f41e-46a3-8fcf-7b94cf50919f","Type":"ContainerDied","Data":"3052b5ff3810b34712b09970122529478ac9632cddcdeb767b46a247b5b02937"} Oct 30 22:20:29.145157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.145147 4933 scope.go:122] "RemoveContainer" containerID="3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65" Oct 30 22:20:29.156817 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.156802 4933 scope.go:122] "RemoveContainer" containerID="3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65" Oct 30 22:20:29.157093 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:20:29.157076 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65\": container with ID starting with 3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65 not found: ID does not exist" containerID="3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65" Oct 30 22:20:29.157153 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.157105 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65"} err="failed to get container status \"3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65\": rpc error: code = NotFound desc = could not find container \"3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65\": container with ID starting with 3f283025b112c16da61c4a7215344a84217b762abdef9bc97edca7c55b027e65 not found: ID does not exist" Oct 30 22:20:29.318593 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.318562 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76"] Oct 30 22:20:29.368273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.368239 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-29cbad-openshift-default-6d74d46b79-5gm76"] Oct 30 22:20:29.466132 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:29.466055 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" path="/var/lib/kubelet/pods/b2bf740a-f41e-46a3-8fcf-7b94cf50919f/volumes" Oct 30 22:20:37.583597 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:37.583556 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:20:40.809022 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:40.808932 4933 scope.go:122] "RemoveContainer" containerID="706abb3d6a1978ff94075525aa447a441234a0f4ea7ccc53e18176d6ee710ba6" Oct 30 22:20:40.821857 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:40.821836 4933 scope.go:122] "RemoveContainer" containerID="96f06bfa9b038f50748892e9f19ec502bb75c7f0725a25c6a89c9500b2ec7927" Oct 30 22:20:40.833788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:40.833764 4933 scope.go:122] "RemoveContainer" containerID="01f7820053c151ab2662ddcb2da78585e67f6b92c50c3739ceb5ee05179e57dd" Oct 30 22:20:48.437929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.437897 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8"] Oct 30 22:20:48.438308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438294 4933 cpu_manager.go:414] "containerMap: removing container" podUID="937b21e4-e2db-4d0b-bd1a-8af39cc2b564" containerName="revoke-keys" Oct 30 22:20:48.438308 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438305 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937b21e4-e2db-4d0b-bd1a-8af39cc2b564" containerName="revoke-keys" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438321 4933 cpu_manager.go:414] "containerMap: removing container" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438327 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438344 4933 cpu_manager.go:414] "containerMap: removing container" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438349 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438358 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" containerName="payload-pre-processing" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438363 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" containerName="payload-pre-processing" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438381 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" Oct 30 22:20:48.438386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438386 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" Oct 30 22:20:48.438684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438442 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bf740a-f41e-46a3-8fcf-7b94cf50919f" containerName="istio-proxy" Oct 30 22:20:48.438684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438449 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e696cd7-f7b8-4da0-bf63-00b0ecd3c266" containerName="payload-pre-processing" Oct 30 22:20:48.438684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438457 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" Oct 30 22:20:48.438684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438463 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0a49b7-dea2-4923-bd0a-30e48455c795" containerName="main" Oct 30 22:20:48.438684 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.438471 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="937b21e4-e2db-4d0b-bd1a-8af39cc2b564" containerName="revoke-keys" Oct 30 22:20:48.443218 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.443202 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.499029 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.499007 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8"] Oct 30 22:20:48.529464 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.529437 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.529613 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.529580 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7c2s\" (UniqueName: \"kubernetes.io/projected/5c3c480e-163a-4117-a39f-4f73cbccde91-kube-api-access-c7c2s\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.529671 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.529655 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.546177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.546155 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:48.546310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.546252 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:48.546310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.546260 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:48.547614 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.547583 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:20:48.630256 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.630233 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.630362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.630280 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7c2s\" (UniqueName: \"kubernetes.io/projected/5c3c480e-163a-4117-a39f-4f73cbccde91-kube-api-access-c7c2s\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.630362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.630330 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.630885 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.630858 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.632804 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.632787 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.688178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.688128 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7c2s\" (UniqueName: \"kubernetes.io/projected/5c3c480e-163a-4117-a39f-4f73cbccde91-kube-api-access-c7c2s\") pod \"maas-api-revoke-sub-138e6235-zfjl8\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:48.756360 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:48.756335 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:49.145144 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:49.145116 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8"] Oct 30 22:20:49.145368 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:20:49.145345 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3c480e_163a_4117_a39f_4f73cbccde91.slice/crio-d18b6020b1133a932401cb85792a30ff7c03e405024fec83a9955ac90fcaa7ad WatchSource:0}: Error finding container d18b6020b1133a932401cb85792a30ff7c03e405024fec83a9955ac90fcaa7ad: Status 404 returned error can't find the container with id d18b6020b1133a932401cb85792a30ff7c03e405024fec83a9955ac90fcaa7ad Oct 30 22:20:49.294875 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:49.294838 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" event={"ID":"5c3c480e-163a-4117-a39f-4f73cbccde91","Type":"ContainerStarted","Data":"d18b6020b1133a932401cb85792a30ff7c03e405024fec83a9955ac90fcaa7ad"} Oct 30 22:20:50.303914 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:50.303884 4933 generic.go:356] "Generic (PLEG): container finished" podID="5c3c480e-163a-4117-a39f-4f73cbccde91" containerID="0bebeba416c431e234e38530c503662151647e88056f2f6e31155248e554b706" exitCode=0 Oct 30 22:20:50.304332 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:50.303930 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" event={"ID":"5c3c480e-163a-4117-a39f-4f73cbccde91","Type":"ContainerDied","Data":"0bebeba416c431e234e38530c503662151647e88056f2f6e31155248e554b706"} Oct 30 22:20:51.455517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.455481 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:51.558111 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.558087 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3c480e-163a-4117-a39f-4f73cbccde91-kube-api-access-c7c2s\" (UniqueName: \"kubernetes.io/projected/5c3c480e-163a-4117-a39f-4f73cbccde91-kube-api-access-c7c2s\") pod \"5c3c480e-163a-4117-a39f-4f73cbccde91\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " Oct 30 22:20:51.558240 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.558229 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-openshift-ca\") pod \"5c3c480e-163a-4117-a39f-4f73cbccde91\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " Oct 30 22:20:51.558291 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.558275 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-xks-ca\") pod \"5c3c480e-163a-4117-a39f-4f73cbccde91\" (UID: \"5c3c480e-163a-4117-a39f-4f73cbccde91\") " Oct 30 22:20:51.558604 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.558581 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-openshift-ca" pod "5c3c480e-163a-4117-a39f-4f73cbccde91" (UID: "5c3c480e-163a-4117-a39f-4f73cbccde91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:51.560335 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.560309 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-xks-ca" pod "5c3c480e-163a-4117-a39f-4f73cbccde91" (UID: "5c3c480e-163a-4117-a39f-4f73cbccde91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:20:51.560440 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.560416 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3c480e-163a-4117-a39f-4f73cbccde91-kube-api-access-c7c2s" pod "5c3c480e-163a-4117-a39f-4f73cbccde91" (UID: "5c3c480e-163a-4117-a39f-4f73cbccde91"). InnerVolumeSpecName "kube-api-access-c7c2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:51.659078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.659051 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7c2s\" (UniqueName: \"kubernetes.io/projected/5c3c480e-163a-4117-a39f-4f73cbccde91-kube-api-access-c7c2s\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:51.659078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.659080 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:51.659273 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:51.659096 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c3c480e-163a-4117-a39f-4f73cbccde91-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:20:52.318053 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:52.318023 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" Oct 30 22:20:52.318053 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:52.318031 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8" event={"ID":"5c3c480e-163a-4117-a39f-4f73cbccde91","Type":"ContainerDied","Data":"d18b6020b1133a932401cb85792a30ff7c03e405024fec83a9955ac90fcaa7ad"} Oct 30 22:20:52.318272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:52.318082 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d18b6020b1133a932401cb85792a30ff7c03e405024fec83a9955ac90fcaa7ad" Oct 30 22:20:58.543433 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:20:58.543396 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:20:59.048359 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.048313 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x"] Oct 30 22:20:59.098760 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.098732 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ebbfe6e-fwt5x"] Oct 30 22:20:59.101610 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.101587 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4"] Oct 30 22:20:59.103762 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.103742 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b161ac3-4zjv4"] Oct 30 22:20:59.464697 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.464623 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5137afc6-0d4c-4cc3-80c0-af971c57aa4c" path="/var/lib/kubelet/pods/5137afc6-0d4c-4cc3-80c0-af971c57aa4c/volumes" Oct 30 22:20:59.465039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.465026 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65fdb8d-a940-4e8c-b589-f4254f5104da" path="/var/lib/kubelet/pods/d65fdb8d-a940-4e8c-b589-f4254f5104da/volumes" Oct 30 22:20:59.607685 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.607660 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2"] Oct 30 22:20:59.608110 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.608093 4933 cpu_manager.go:414] "containerMap: removing container" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" Oct 30 22:20:59.608154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.608111 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" Oct 30 22:20:59.608154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.608133 4933 cpu_manager.go:414] "containerMap: removing container" podUID="5c3c480e-163a-4117-a39f-4f73cbccde91" containerName="revoke-keys" Oct 30 22:20:59.608154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.608138 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3c480e-163a-4117-a39f-4f73cbccde91" containerName="revoke-keys" Oct 30 22:20:59.608268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.608215 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="386b34f2-8410-4621-9678-4c7a1df8270f" containerName="istio-proxy" Oct 30 22:20:59.608268 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.608230 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3c480e-163a-4117-a39f-4f73cbccde91" containerName="revoke-keys" Oct 30 22:20:59.611584 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.611567 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.668274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.668250 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2"] Oct 30 22:20:59.715425 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.715363 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:20:59.715639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.715453 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:20:59.715639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.715468 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:20:59.715639 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.715566 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:20:59.730211 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.730186 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvk2\" (UniqueName: \"kubernetes.io/projected/43c4ae03-61bc-45b9-a440-60ea9bd4773c-kube-api-access-fjvk2\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.730334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.730218 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.730334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.730241 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.831439 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.831409 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvk2\" (UniqueName: \"kubernetes.io/projected/43c4ae03-61bc-45b9-a440-60ea9bd4773c-kube-api-access-fjvk2\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.831622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.831445 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.831622 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.831470 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.832058 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.832034 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.833958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.833941 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.889618 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.889599 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvk2\" (UniqueName: \"kubernetes.io/projected/43c4ae03-61bc-45b9-a440-60ea9bd4773c-kube-api-access-fjvk2\") pod \"maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:20:59.924734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:20:59.924714 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:21:00.299436 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:00.299408 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2"] Oct 30 22:21:00.299698 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:21:00.299676 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c4ae03_61bc_45b9_a440_60ea9bd4773c.slice/crio-37f9ddf5d5f87c20564d7343ba1a457bf8a60c5a3482644ddabb4f852405501d WatchSource:0}: Error finding container 37f9ddf5d5f87c20564d7343ba1a457bf8a60c5a3482644ddabb4f852405501d: Status 404 returned error can't find the container with id 37f9ddf5d5f87c20564d7343ba1a457bf8a60c5a3482644ddabb4f852405501d Oct 30 22:21:00.371244 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:00.371218 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" event={"ID":"43c4ae03-61bc-45b9-a440-60ea9bd4773c","Type":"ContainerStarted","Data":"37f9ddf5d5f87c20564d7343ba1a457bf8a60c5a3482644ddabb4f852405501d"} Oct 30 22:21:01.379681 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:01.379637 4933 generic.go:356] "Generic (PLEG): container finished" podID="43c4ae03-61bc-45b9-a440-60ea9bd4773c" containerID="8f3c099d11fb5ca97d7b0af4e4cf7adde4160b8356a5e25303758f00b6aa9d31" exitCode=0 Oct 30 22:21:01.380062 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:01.379688 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" event={"ID":"43c4ae03-61bc-45b9-a440-60ea9bd4773c","Type":"ContainerDied","Data":"8f3c099d11fb5ca97d7b0af4e4cf7adde4160b8356a5e25303758f00b6aa9d31"} Oct 30 22:21:02.522324 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.522301 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:21:02.718451 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.718357 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c4ae03-61bc-45b9-a440-60ea9bd4773c-kube-api-access-fjvk2\" (UniqueName: \"kubernetes.io/projected/43c4ae03-61bc-45b9-a440-60ea9bd4773c-kube-api-access-fjvk2\") pod \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " Oct 30 22:21:02.719401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.719366 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-openshift-ca\") pod \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " Oct 30 22:21:02.721041 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.721004 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-xks-ca\") pod \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\" (UID: \"43c4ae03-61bc-45b9-a440-60ea9bd4773c\") " Oct 30 22:21:02.721866 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.719843 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-openshift-ca" pod "43c4ae03-61bc-45b9-a440-60ea9bd4773c" (UID: "43c4ae03-61bc-45b9-a440-60ea9bd4773c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:02.722673 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.722606 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c4ae03-61bc-45b9-a440-60ea9bd4773c-kube-api-access-fjvk2" pod "43c4ae03-61bc-45b9-a440-60ea9bd4773c" (UID: "43c4ae03-61bc-45b9-a440-60ea9bd4773c"). InnerVolumeSpecName "kube-api-access-fjvk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:02.785648 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.785604 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-xks-ca" pod "43c4ae03-61bc-45b9-a440-60ea9bd4773c" (UID: "43c4ae03-61bc-45b9-a440-60ea9bd4773c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:02.822816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.822785 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjvk2\" (UniqueName: \"kubernetes.io/projected/43c4ae03-61bc-45b9-a440-60ea9bd4773c-kube-api-access-fjvk2\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:02.822816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.822807 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:02.822816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:02.822817 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4ae03-61bc-45b9-a440-60ea9bd4773c-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:03.413526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:03.413324 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" event={"ID":"43c4ae03-61bc-45b9-a440-60ea9bd4773c","Type":"ContainerDied","Data":"37f9ddf5d5f87c20564d7343ba1a457bf8a60c5a3482644ddabb4f852405501d"} Oct 30 22:21:03.413526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:03.413385 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f9ddf5d5f87c20564d7343ba1a457bf8a60c5a3482644ddabb4f852405501d" Oct 30 22:21:03.413526 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:03.413452 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2" Oct 30 22:21:04.054515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:04.054210 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq"] Oct 30 22:21:04.107517 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:04.103852 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b9db884-c6fmq"] Oct 30 22:21:04.117008 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:04.116897 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl"] Oct 30 22:21:04.120795 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:04.120741 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr"] Oct 30 22:21:05.465814 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:05.465780 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e01147-e64d-4c5e-b2e6-27d18a84e378" path="/var/lib/kubelet/pods/89e01147-e64d-4c5e-b2e6-27d18a84e378/volumes" Oct 30 22:21:14.926636 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:14.926605 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp"] Oct 30 22:21:14.927234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:14.927203 4933 cpu_manager.go:414] "containerMap: removing container" podUID="43c4ae03-61bc-45b9-a440-60ea9bd4773c" containerName="revoke-keys" Oct 30 22:21:14.927234 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:14.927221 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c4ae03-61bc-45b9-a440-60ea9bd4773c" containerName="revoke-keys" Oct 30 22:21:14.927386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:14.927371 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c4ae03-61bc-45b9-a440-60ea9bd4773c" containerName="revoke-keys" Oct 30 22:21:14.931720 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:14.931698 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.025435 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.025410 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:21:15.025683 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.025459 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:21:15.025683 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.025527 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:21:15.028716 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.028694 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp"] Oct 30 22:21:15.033145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.033125 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:21:15.058199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.058168 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/500f517d-042b-4a76-9c76-86e18a298464-kube-api-access-8t9wk\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.058322 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.058201 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500f517d-042b-4a76-9c76-86e18a298464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.058391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.058365 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500f517d-042b-4a76-9c76-86e18a298464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.159480 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.159442 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500f517d-042b-4a76-9c76-86e18a298464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.159623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.159559 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/500f517d-042b-4a76-9c76-86e18a298464-kube-api-access-8t9wk\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.159623 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.159584 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500f517d-042b-4a76-9c76-86e18a298464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.160100 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.160073 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500f517d-042b-4a76-9c76-86e18a298464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.162122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.162097 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500f517d-042b-4a76-9c76-86e18a298464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.217367 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.217312 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/500f517d-042b-4a76-9c76-86e18a298464-kube-api-access-8t9wk\") pod \"maas-api-revoke-sub-428fbc37-8rnwp\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.245207 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.245189 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:15.623157 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:15.623127 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp"] Oct 30 22:21:15.623613 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:21:15.623590 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500f517d_042b_4a76_9c76_86e18a298464.slice/crio-48942faf472894eabbfbcf675bbd9e60a513e077255abea338f3a4cd6bd5b581 WatchSource:0}: Error finding container 48942faf472894eabbfbcf675bbd9e60a513e077255abea338f3a4cd6bd5b581: Status 404 returned error can't find the container with id 48942faf472894eabbfbcf675bbd9e60a513e077255abea338f3a4cd6bd5b581 Oct 30 22:21:16.510573 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:16.510540 4933 generic.go:356] "Generic (PLEG): container finished" podID="500f517d-042b-4a76-9c76-86e18a298464" containerID="4d3038ba1079758c92ab78441606e6d3c8d3eac79f9ddf0ecdf75c820d22e36f" exitCode=0 Oct 30 22:21:16.510973 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:16.510626 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" event={"ID":"500f517d-042b-4a76-9c76-86e18a298464","Type":"ContainerDied","Data":"4d3038ba1079758c92ab78441606e6d3c8d3eac79f9ddf0ecdf75c820d22e36f"} Oct 30 22:21:16.510973 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:16.510697 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" event={"ID":"500f517d-042b-4a76-9c76-86e18a298464","Type":"ContainerStarted","Data":"48942faf472894eabbfbcf675bbd9e60a513e077255abea338f3a4cd6bd5b581"} Oct 30 22:21:17.657125 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.657104 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:17.782545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.782516 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500f517d-042b-4a76-9c76-86e18a298464-kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/500f517d-042b-4a76-9c76-86e18a298464-kube-api-access-8t9wk\") pod \"500f517d-042b-4a76-9c76-86e18a298464\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " Oct 30 22:21:17.782701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.782586 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500f517d-042b-4a76-9c76-86e18a298464-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500f517d-042b-4a76-9c76-86e18a298464-maas-api-openshift-ca\") pod \"500f517d-042b-4a76-9c76-86e18a298464\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " Oct 30 22:21:17.782701 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.782687 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500f517d-042b-4a76-9c76-86e18a298464-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500f517d-042b-4a76-9c76-86e18a298464-maas-api-xks-ca\") pod \"500f517d-042b-4a76-9c76-86e18a298464\" (UID: \"500f517d-042b-4a76-9c76-86e18a298464\") " Oct 30 22:21:17.783001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.782975 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500f517d-042b-4a76-9c76-86e18a298464-maas-api-openshift-ca" pod "500f517d-042b-4a76-9c76-86e18a298464" (UID: "500f517d-042b-4a76-9c76-86e18a298464"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:17.783195 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.783110 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500f517d-042b-4a76-9c76-86e18a298464-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:17.784679 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.784654 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500f517d-042b-4a76-9c76-86e18a298464-kube-api-access-8t9wk" pod "500f517d-042b-4a76-9c76-86e18a298464" (UID: "500f517d-042b-4a76-9c76-86e18a298464"). InnerVolumeSpecName "kube-api-access-8t9wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:17.784813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.784792 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500f517d-042b-4a76-9c76-86e18a298464-maas-api-xks-ca" pod "500f517d-042b-4a76-9c76-86e18a298464" (UID: "500f517d-042b-4a76-9c76-86e18a298464"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:17.884537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.884473 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500f517d-042b-4a76-9c76-86e18a298464-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:17.884537 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:17.884530 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9wk\" (UniqueName: \"kubernetes.io/projected/500f517d-042b-4a76-9c76-86e18a298464-kube-api-access-8t9wk\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:18.525508 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:18.525457 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" Oct 30 22:21:18.525508 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:18.525479 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp" event={"ID":"500f517d-042b-4a76-9c76-86e18a298464","Type":"ContainerDied","Data":"48942faf472894eabbfbcf675bbd9e60a513e077255abea338f3a4cd6bd5b581"} Oct 30 22:21:18.525731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:18.525549 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48942faf472894eabbfbcf675bbd9e60a513e077255abea338f3a4cd6bd5b581" Oct 30 22:21:19.148032 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.147972 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" containerID="cri-o://b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e" gracePeriod=45 Oct 30 22:21:19.158920 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.158889 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" containerID="cri-o://8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6" gracePeriod=45 Oct 30 22:21:19.431145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.431120 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:21:19.460179 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.460154 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:21:19.500401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500369 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-model-cache\") pod \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " Oct 30 22:21:19.500612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500424 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tls-certs\") pod \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " Oct 30 22:21:19.500612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500470 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tmp-dir\") pod \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " Oct 30 22:21:19.500612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500557 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-dshm\") pod \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " Oct 30 22:21:19.500791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500676 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-home\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-home\") pod \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " Oct 30 22:21:19.500791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500696 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-model-cache" pod "cc0ca645-0bf9-4e47-9cd3-222af475fff6" (UID: "cc0ca645-0bf9-4e47-9cd3-222af475fff6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.500791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500713 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tmp-dir" pod "cc0ca645-0bf9-4e47-9cd3-222af475fff6" (UID: "cc0ca645-0bf9-4e47-9cd3-222af475fff6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.500791 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500739 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0ca645-0bf9-4e47-9cd3-222af475fff6-kube-api-access-hhgvt\" (UniqueName: \"kubernetes.io/projected/cc0ca645-0bf9-4e47-9cd3-222af475fff6-kube-api-access-hhgvt\") pod \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\" (UID: \"cc0ca645-0bf9-4e47-9cd3-222af475fff6\") " Oct 30 22:21:19.501030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.500912 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-home" pod "cc0ca645-0bf9-4e47-9cd3-222af475fff6" (UID: "cc0ca645-0bf9-4e47-9cd3-222af475fff6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.501129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.501103 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.501190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.501137 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.501190 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.501150 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.503249 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.503222 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0ca645-0bf9-4e47-9cd3-222af475fff6-kube-api-access-hhgvt" pod "cc0ca645-0bf9-4e47-9cd3-222af475fff6" (UID: "cc0ca645-0bf9-4e47-9cd3-222af475fff6"). InnerVolumeSpecName "kube-api-access-hhgvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:19.503394 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.503292 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tls-certs" pod "cc0ca645-0bf9-4e47-9cd3-222af475fff6" (UID: "cc0ca645-0bf9-4e47-9cd3-222af475fff6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:19.503394 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.503290 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-dshm" pod "cc0ca645-0bf9-4e47-9cd3-222af475fff6" (UID: "cc0ca645-0bf9-4e47-9cd3-222af475fff6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.535199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.535164 4933 generic.go:356] "Generic (PLEG): container finished" podID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerID="b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e" exitCode=0 Oct 30 22:21:19.535381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.535204 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" Oct 30 22:21:19.535381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.535256 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" event={"ID":"cc0ca645-0bf9-4e47-9cd3-222af475fff6","Type":"ContainerDied","Data":"b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e"} Oct 30 22:21:19.535381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.535294 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr" event={"ID":"cc0ca645-0bf9-4e47-9cd3-222af475fff6","Type":"ContainerDied","Data":"0cb6c15c162ffbe0bedf641395f295b26ec3c261e1d1bc25ba08f21ebea9418d"} Oct 30 22:21:19.535381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.535314 4933 scope.go:122] "RemoveContainer" containerID="b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e" Oct 30 22:21:19.537037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.537011 4933 generic.go:356] "Generic (PLEG): container finished" podID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerID="8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6" exitCode=0 Oct 30 22:21:19.537184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.537054 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" Oct 30 22:21:19.537184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.537088 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" event={"ID":"0d6faa2f-6fd1-43cd-8d36-d020bab4b638","Type":"ContainerDied","Data":"8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6"} Oct 30 22:21:19.537184 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.537126 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl" event={"ID":"0d6faa2f-6fd1-43cd-8d36-d020bab4b638","Type":"ContainerDied","Data":"a51c2dfcf93cb907f8d577fe8f203ca7448b0bea18745fd1dda814401b2981f5"} Oct 30 22:21:19.555734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.553508 4933 scope.go:122] "RemoveContainer" containerID="b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e" Oct 30 22:21:19.555734 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:21:19.553966 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e\": container with ID starting with b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e not found: ID does not exist" containerID="b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e" Oct 30 22:21:19.555734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.554003 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e"} err="failed to get container status \"b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e\": rpc error: code = NotFound desc = could not find container \"b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e\": container with ID starting with b15577a73cd2406125f58fd4dc022f35f5290ec3f24145e949fa79c75ec1bb8e not found: ID does not exist" Oct 30 22:21:19.555734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.554028 4933 scope.go:122] "RemoveContainer" containerID="8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6" Oct 30 22:21:19.566220 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.566158 4933 scope.go:122] "RemoveContainer" containerID="8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6" Oct 30 22:21:19.566713 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:21:19.566640 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6\": container with ID starting with 8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6 not found: ID does not exist" containerID="8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6" Oct 30 22:21:19.566713 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.566678 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6"} err="failed to get container status \"8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6\": rpc error: code = NotFound desc = could not find container \"8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6\": container with ID starting with 8010442f60835e7eb1dc68824eeaf47962f013b200f34167d9263f611863cfb6 not found: ID does not exist" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.602353 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-dshm\") pod \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.602409 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-model-cache\") pod \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.602477 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-home\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-home\") pod \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.602588 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tls-certs\") pod \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.602630 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-kube-api-access-bs68h\" (UniqueName: \"kubernetes.io/projected/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-kube-api-access-bs68h\") pod \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.602666 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tmp-dir\") pod \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\" (UID: \"0d6faa2f-6fd1-43cd-8d36-d020bab4b638\") " Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.603080 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-home" pod "0d6faa2f-6fd1-43cd-8d36-d020bab4b638" (UID: "0d6faa2f-6fd1-43cd-8d36-d020bab4b638"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.603723 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tmp-dir" pod "0d6faa2f-6fd1-43cd-8d36-d020bab4b638" (UID: "0d6faa2f-6fd1-43cd-8d36-d020bab4b638"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.604090 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-model-cache" pod "0d6faa2f-6fd1-43cd-8d36-d020bab4b638" (UID: "0d6faa2f-6fd1-43cd-8d36-d020bab4b638"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.604561 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhgvt\" (UniqueName: \"kubernetes.io/projected/cc0ca645-0bf9-4e47-9cd3-222af475fff6-kube-api-access-hhgvt\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.604585 4933 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tmp-dir\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.604606 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc0ca645-0bf9-4e47-9cd3-222af475fff6-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.604631 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc0ca645-0bf9-4e47-9cd3-222af475fff6-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.604645 4933 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-home\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.607317 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tls-certs" pod "0d6faa2f-6fd1-43cd-8d36-d020bab4b638" (UID: "0d6faa2f-6fd1-43cd-8d36-d020bab4b638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:21:19.608528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.608025 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-kube-api-access-bs68h" pod "0d6faa2f-6fd1-43cd-8d36-d020bab4b638" (UID: "0d6faa2f-6fd1-43cd-8d36-d020bab4b638"). InnerVolumeSpecName "kube-api-access-bs68h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:19.612731 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.612706 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-dshm" pod "0d6faa2f-6fd1-43cd-8d36-d020bab4b638" (UID: "0d6faa2f-6fd1-43cd-8d36-d020bab4b638"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:19.705267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.705192 4933 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-tls-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.705267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.705220 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs68h\" (UniqueName: \"kubernetes.io/projected/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-kube-api-access-bs68h\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.705267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.705231 4933 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-dshm\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.705267 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.705239 4933 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d6faa2f-6fd1-43cd-8d36-d020bab4b638-model-cache\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:19.837044 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.837012 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr"] Oct 30 22:21:19.843793 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.843765 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/simulator-w3-0f42cb-kserve-5c799cb66f-4t5lr"] Oct 30 22:21:19.962793 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.962701 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl"] Oct 30 22:21:19.966104 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:19.966075 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0f42cb/premium-w3-0f42cb-kserve-cbdfcc77b-zwgzl"] Oct 30 22:21:20.049785 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:20.049757 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls"] Oct 30 22:21:20.052528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:20.052482 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28e69f00-wr9ls"] Oct 30 22:21:20.056038 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:20.056015 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts"] Oct 30 22:21:20.059824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:20.059802 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd8213aa-ctvts"] Oct 30 22:21:21.468294 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:21.468260 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" path="/var/lib/kubelet/pods/0d6faa2f-6fd1-43cd-8d36-d020bab4b638/volumes" Oct 30 22:21:21.468913 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:21.468893 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67346a3-27eb-41a9-871a-9947ad851b1a" path="/var/lib/kubelet/pods/b67346a3-27eb-41a9-871a-9947ad851b1a/volumes" Oct 30 22:21:21.469453 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:21.469434 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" path="/var/lib/kubelet/pods/cc0ca645-0bf9-4e47-9cd3-222af475fff6/volumes" Oct 30 22:21:21.470305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:21.470290 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcd3373-7782-4c15-a862-12c28d631416" path="/var/lib/kubelet/pods/cfcd3373-7782-4c15-a862-12c28d631416/volumes" Oct 30 22:21:28.543351 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:21:28.543314 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:21:31.863911 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:31.863868 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:21:32.389349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:32.389317 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2"] Oct 30 22:21:32.439899 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:32.439873 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-0f42cb-58mr2"] Oct 30 22:21:33.464547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:33.464454 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c4ae03-61bc-45b9-a440-60ea9bd4773c" path="/var/lib/kubelet/pods/43c4ae03-61bc-45b9-a440-60ea9bd4773c/volumes" Oct 30 22:21:39.500938 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:39.500898 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q"] Oct 30 22:21:39.501403 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:39.501098 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" containerID="cri-o://237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b" gracePeriod=30 Oct 30 22:21:39.518010 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:39.517977 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89"] Oct 30 22:21:39.518239 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:39.518189 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" containerID="cri-o://5e2aa04f2501f4d8a9037f44a91c195aa9f71392f09dedf9dd3a6f38e5793678" gracePeriod=30 Oct 30 22:21:40.893177 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:40.893149 4933 scope.go:122] "RemoveContainer" containerID="016412c5c9a2d10d28d34e03f456d5ad1dd8f0b2a0d09ff13ef12c44c0d90fcd" Oct 30 22:21:40.911087 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:40.911060 4933 scope.go:122] "RemoveContainer" containerID="456f3c0597c47b86fd25ce5c16095b7a16f6d893211a4be409262c2ae92473cd" Oct 30 22:21:40.923811 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:40.923701 4933 scope.go:122] "RemoveContainer" containerID="7710609c1c3ddfd4d38d485b89e082359f1d89f73ccf642a25fb1ad19154ab97" Oct 30 22:21:44.781939 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.781911 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:21:44.798665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.798638 4933 generic.go:356] "Generic (PLEG): container finished" podID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerID="5e2aa04f2501f4d8a9037f44a91c195aa9f71392f09dedf9dd3a6f38e5793678" exitCode=0 Oct 30 22:21:44.798787 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.798711 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" event={"ID":"dcf04c1d-4538-4574-aa8c-6b44e958e092","Type":"ContainerDied","Data":"5e2aa04f2501f4d8a9037f44a91c195aa9f71392f09dedf9dd3a6f38e5793678"} Oct 30 22:21:44.800345 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.800316 4933 generic.go:356] "Generic (PLEG): container finished" podID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerID="237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b" exitCode=0 Oct 30 22:21:44.800481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.800379 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" Oct 30 22:21:44.800481 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.800470 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" event={"ID":"d3ddd21c-c122-4046-9804-c3bcd2b34bc3","Type":"ContainerDied","Data":"237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b"} Oct 30 22:21:44.800652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.800544 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q" event={"ID":"d3ddd21c-c122-4046-9804-c3bcd2b34bc3","Type":"ContainerDied","Data":"8e93f49494c41224544fb6d4529aed2e47b7f1558eca88f49cb404b790892f8b"} Oct 30 22:21:44.800652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.800580 4933 scope.go:122] "RemoveContainer" containerID="237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b" Oct 30 22:21:44.816997 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.816981 4933 scope.go:122] "RemoveContainer" containerID="237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b" Oct 30 22:21:44.817291 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:21:44.817272 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b\": container with ID starting with 237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b not found: ID does not exist" containerID="237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b" Oct 30 22:21:44.817349 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.817300 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b"} err="failed to get container status \"237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b\": rpc error: code = NotFound desc = could not find container \"237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b\": container with ID starting with 237f2b70d73f916f289afa5dd7b9128567d1f44e23b0967d3c15c05d2d741c5b not found: ID does not exist" Oct 30 22:21:44.818522 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.818507 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:21:44.913587 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913558 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-certs\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.913734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913614 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-data\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.913734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913641 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-envoy\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.913734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913661 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-token\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-token\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.913734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913685 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf04c1d-4538-4574-aa8c-6b44e958e092-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf04c1d-4538-4574-aa8c-6b44e958e092-istiod-ca-cert\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.913734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913704 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-credential-socket\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.913734 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913723 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-data\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.914078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913884 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-data" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.914078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913895 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-certs" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.914078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.913999 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-credential-socket" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.914078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914051 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-data" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914084 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istiod-ca-cert\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914131 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-certs\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914171 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-token\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-token\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914221 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-podinfo\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914273 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-kube-api-access-tfjxv\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914303 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-socket\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914329 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-kube-api-access-thmmz\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-kube-api-access-thmmz\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914372 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-socket\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.914344 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914406 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-podinfo\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914430 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-certs" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914451 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-envoy\") pod \"dcf04c1d-4538-4574-aa8c-6b44e958e092\" (UID: \"dcf04c1d-4538-4574-aa8c-6b44e958e092\") " Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914471 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istiod-ca-cert" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914520 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-credential-socket\") pod \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\" (UID: \"d3ddd21c-c122-4046-9804-c3bcd2b34bc3\") " Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914642 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-socket" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914715 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-socket" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914889 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914909 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914923 4933 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-workload-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914938 4933 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-workload-certs\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914951 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914965 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914978 4933 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-data\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.914992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.914992 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:44.916250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.915057 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf04c1d-4538-4574-aa8c-6b44e958e092-istiod-ca-cert" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:44.916250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.915442 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-credential-socket" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.917258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.917223 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-token" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:44.917366 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.917337 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-envoy" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.918570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.918211 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-envoy" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:44.919588 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.919560 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-token" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:44.919706 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.919636 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-kube-api-access-tfjxv" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "kube-api-access-tfjxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:44.919789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.919755 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-podinfo" pod "d3ddd21c-c122-4046-9804-c3bcd2b34bc3" (UID: "d3ddd21c-c122-4046-9804-c3bcd2b34bc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:44.920347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.920282 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-kube-api-access-thmmz" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "kube-api-access-thmmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:44.920450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:44.920374 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-podinfo" pod "dcf04c1d-4538-4574-aa8c-6b44e958e092" (UID: "dcf04c1d-4538-4574-aa8c-6b44e958e092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:45.015680 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015651 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015680 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015675 4933 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-credential-socket\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015680 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015686 4933 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-envoy\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015696 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015706 4933 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf04c1d-4538-4574-aa8c-6b44e958e092-istiod-ca-cert\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015715 4933 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-token\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015723 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015732 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfjxv\" (UniqueName: \"kubernetes.io/projected/d3ddd21c-c122-4046-9804-c3bcd2b34bc3-kube-api-access-tfjxv\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015741 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmmz\" (UniqueName: \"kubernetes.io/projected/dcf04c1d-4538-4574-aa8c-6b44e958e092-kube-api-access-thmmz\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.015921 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.015748 4933 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf04c1d-4538-4574-aa8c-6b44e958e092-istio-podinfo\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:21:45.249760 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.249728 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q"] Oct 30 22:21:45.256655 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.256623 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-pbn4q"] Oct 30 22:21:45.464929 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.464853 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" path="/var/lib/kubelet/pods/d3ddd21c-c122-4046-9804-c3bcd2b34bc3/volumes" Oct 30 22:21:45.807531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.807476 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" Oct 30 22:21:45.807531 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.807475 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89" event={"ID":"dcf04c1d-4538-4574-aa8c-6b44e958e092","Type":"ContainerDied","Data":"f94a3427b701ddaffb90f16a1add66ece24e70879accfae1013b0693d9c0ddf3"} Oct 30 22:21:45.808082 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.807600 4933 scope.go:122] "RemoveContainer" containerID="5e2aa04f2501f4d8a9037f44a91c195aa9f71392f09dedf9dd3a6f38e5793678" Oct 30 22:21:45.974775 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:45.974748 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89"] Oct 30 22:21:46.025628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:46.025604 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0f42cb-openshift-default-584f4c4865-6vd89"] Oct 30 22:21:47.067434 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:47.067399 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:21:47.468407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:21:47.468318 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" path="/var/lib/kubelet/pods/dcf04c1d-4538-4574-aa8c-6b44e958e092/volumes" Oct 30 22:21:58.543242 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:21:58.543204 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:22:12.051854 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:12.051781 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp"] Oct 30 22:22:12.102310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:12.102283 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539a2043-xjchp"] Oct 30 22:22:12.104355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:12.104332 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2"] Oct 30 22:22:12.106197 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:12.106178 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0abedc8a-vfxm2"] Oct 30 22:22:13.471878 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:13.471842 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c75fcfd-fac3-45d6-a99c-9519c793396e" path="/var/lib/kubelet/pods/0c75fcfd-fac3-45d6-a99c-9519c793396e/volumes" Oct 30 22:22:13.472258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:13.472231 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f58c787-5475-4f57-ae80-2ece9ae38bda" path="/var/lib/kubelet/pods/7f58c787-5475-4f57-ae80-2ece9ae38bda/volumes" Oct 30 22:22:28.543231 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:22:28.543197 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:22:31.948314 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:31.948278 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:22:31.951601 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:31.951580 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:22:40.985013 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:40.984978 4933 scope.go:122] "RemoveContainer" containerID="5570f8988f4fc443b40dc79d8f875e2468ded5f43331294a94f4d972a84b8606" Oct 30 22:22:40.994626 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:40.994593 4933 scope.go:122] "RemoveContainer" containerID="629dfa400845e9e75e1143271a5d50d90d2f4b61d81c7875c7c6a0e546b28bbd" Oct 30 22:22:41.003547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.003529 4933 scope.go:122] "RemoveContainer" containerID="14927c04e9990508cd70078b58ef9feaeab0434939a7c154aceff3176bce6739" Oct 30 22:22:41.011603 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.011581 4933 scope.go:122] "RemoveContainer" containerID="23155714b710e75be9cd3de626b0c2dbbedbc2942ca40d0b003971982af602c1" Oct 30 22:22:41.019883 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.019870 4933 scope.go:122] "RemoveContainer" containerID="2bf145195bed37e85f10986656d1333bb70752c72462d61538d3eafa06ddba45" Oct 30 22:22:41.027922 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.027902 4933 scope.go:122] "RemoveContainer" containerID="4f5b73ea8149d09510609921b79e800abe41663648ef0eab307e3505a5561a23" Oct 30 22:22:41.035915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.035898 4933 scope.go:122] "RemoveContainer" containerID="1cfb1d067705327c3719cc8672732a80bb2923077bf33b986842519617640bca" Oct 30 22:22:41.043953 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.043936 4933 scope.go:122] "RemoveContainer" containerID="835ce065944f4bf69f5425d2a898c7d550a4d7418dd7d693fa9706bd441bc389" Oct 30 22:22:41.052284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.052267 4933 scope.go:122] "RemoveContainer" containerID="7fa71253ad89140d9bffe95a48aaf5aaa465c1bc4d977391c239554786da1ab4" Oct 30 22:22:41.059882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:22:41.059866 4933 scope.go:122] "RemoveContainer" containerID="e7ae9d7656f5ce123d3457831b8453e82ddf5cadd4588e605bfba8e73c5b9c81" Oct 30 22:22:58.543192 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:22:58.543152 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:23:07.074091 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:07.074052 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:23:28.460867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:28.460838 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:23:28.542939 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:23:28.542904 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:23:38.115771 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:38.115734 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5"] Oct 30 22:23:38.117831 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:38.117809 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv"] Oct 30 22:23:38.145359 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:38.145322 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2418f41-cp7l5"] Oct 30 22:23:38.147291 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:38.147267 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:23:38.165364 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:38.165343 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1683b9-mrcnv"] Oct 30 22:23:39.092127 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:39.092097 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs"] Oct 30 22:23:39.094040 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:39.094022 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c0b420-6jsxs"] Oct 30 22:23:39.464773 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:39.464692 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7b3edf-3e86-427f-8905-62cd07310921" path="/var/lib/kubelet/pods/1e7b3edf-3e86-427f-8905-62cd07310921/volumes" Oct 30 22:23:39.465160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:39.465076 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83e2764-b720-486e-a40d-4cff6193f5df" path="/var/lib/kubelet/pods/e83e2764-b720-486e-a40d-4cff6193f5df/volumes" Oct 30 22:23:39.465440 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:39.465426 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef45a6fb-f758-4028-be3a-5813801497eb" path="/var/lib/kubelet/pods/ef45a6fb-f758-4028-be3a-5813801497eb/volumes" Oct 30 22:23:41.128724 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:41.128656 4933 scope.go:122] "RemoveContainer" containerID="d4defa57fa7e8ff7815f69c03aecc24e4a0ffdb17e342bc925b22212cf92f1fd" Oct 30 22:23:41.138001 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:41.137978 4933 scope.go:122] "RemoveContainer" containerID="b1d045ef1457cc73f4363950cf6fb2d9cf880f892f472f905c35c9c089a5bff6" Oct 30 22:23:41.146013 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:41.145995 4933 scope.go:122] "RemoveContainer" containerID="2bc6908114a8f97d508773bc32c30fc1953b7aeccf431e1671146d73d85daad4" Oct 30 22:23:41.153939 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:41.153918 4933 scope.go:122] "RemoveContainer" containerID="abcf0b978f629712a9bf1c5ec1c68555e6ee12d1dc9ece3d4c7f90ebcf2dbde9" Oct 30 22:23:41.161520 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:23:41.161478 4933 scope.go:122] "RemoveContainer" containerID="2e73c3eb0fc11a2a34ee4f332074416841b63631076ed05e6154ecbbe9fb49c7" Oct 30 22:23:58.543239 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:23:58.543190 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:24:03.478882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:03.478855 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:24:16.680605 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:16.680572 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:24:28.548055 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:24:28.548016 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:24:41.198037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:41.198004 4933 scope.go:122] "RemoveContainer" containerID="6754d2b93ad0714cb9890c68be9f8f2deab7eea5f9409af254f6485bbe735493" Oct 30 22:24:41.209401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:41.209376 4933 scope.go:122] "RemoveContainer" containerID="80897248ce645d5c3353edfb11dfb54abd49464f58af8bbc0b26e8fd0b94a93a" Oct 30 22:24:41.218030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:41.218009 4933 scope.go:122] "RemoveContainer" containerID="88d37aa9426fdfd071dd0883cdb00b4b74716666dcffe8ccb194b8513009d899" Oct 30 22:24:41.225936 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:41.225919 4933 scope.go:122] "RemoveContainer" containerID="6dcdbfdd56429346bf07160a56cced34eeae2342d8deb11c552f0c34130555f4" Oct 30 22:24:41.233683 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:41.233635 4933 scope.go:122] "RemoveContainer" containerID="2567fc5f60ba9b3f3076865eb2c63c591277722e0362c72e04d79d1ee3e18f8c" Oct 30 22:24:41.242013 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:41.241996 4933 scope.go:122] "RemoveContainer" containerID="615e8e67c6d23aad2eec3f371008b73248e5c7e0b27a38bc53cf33a05d0af6e4" Oct 30 22:24:47.774881 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:47.774850 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:24:58.286053 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:24:58.286025 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:24:58.548860 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:24:58.548765 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:25:03.051997 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:03.051964 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8"] Oct 30 22:25:03.102644 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:03.102607 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9f33d-rqfz8"] Oct 30 22:25:03.464142 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:03.464049 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937b21e4-e2db-4d0b-bd1a-8af39cc2b564" path="/var/lib/kubelet/pods/937b21e4-e2db-4d0b-bd1a-8af39cc2b564/volumes" Oct 30 22:25:10.859992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:10.859919 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:25:28.543273 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:25:28.543237 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:25:33.492477 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:33.492402 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:25:41.360752 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:41.358666 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:25:52.046803 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:52.046770 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8"] Oct 30 22:25:52.096846 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:52.096816 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-138e6235-zfjl8"] Oct 30 22:25:53.463740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:25:53.463707 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3c480e-163a-4117-a39f-4f73cbccde91" path="/var/lib/kubelet/pods/5c3c480e-163a-4117-a39f-4f73cbccde91/volumes" Oct 30 22:25:58.542835 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:25:58.542798 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:26:01.487237 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:01.487206 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:26:13.754693 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:13.754659 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:26:18.049999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:18.049967 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp"] Oct 30 22:26:18.099971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:18.099935 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-428fbc37-8rnwp"] Oct 30 22:26:19.464736 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:19.464698 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500f517d-042b-4a76-9c76-86e18a298464" path="/var/lib/kubelet/pods/500f517d-042b-4a76-9c76-86e18a298464/volumes" Oct 30 22:26:28.542966 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:26:28.542928 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:26:33.887223 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:33.887188 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:26:41.288805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:41.288728 4933 scope.go:122] "RemoveContainer" containerID="3d605b3c0c32fc7b09afed9cdeceec44fe6aa99c29c33cf588c9f1493fe6b040" Oct 30 22:26:41.297708 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:41.297693 4933 scope.go:122] "RemoveContainer" containerID="d78243ab5b46564bd4b099d91798a10bbab990f031bebf0837adb86d2e9ff7df" Oct 30 22:26:52.870950 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:26:52.870912 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:26:58.542642 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:26:58.542602 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:27:05.890876 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:27:05.890846 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:27:28.542892 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:27:28.542850 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:27:31.986415 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:27:31.986385 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:27:31.990212 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:27:31.990185 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:27:40.087754 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:27:40.087717 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:27:41.342235 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:27:41.342202 4933 scope.go:122] "RemoveContainer" containerID="0bebeba416c431e234e38530c503662151647e88056f2f6e31155248e554b706" Oct 30 22:27:41.409915 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:27:41.409572 4933 scope.go:122] "RemoveContainer" containerID="8f3c099d11fb5ca97d7b0af4e4cf7adde4160b8356a5e25303758f00b6aa9d31" Oct 30 22:27:41.421861 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:27:41.421841 4933 scope.go:122] "RemoveContainer" containerID="4d3038ba1079758c92ab78441606e6d3c8d3eac79f9ddf0ecdf75c820d22e36f" Oct 30 22:27:58.542975 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:27:58.542934 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:28:03.170407 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:28:03.170373 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:28:13.857715 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:28:13.857622 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:28:28.542584 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:28:28.542549 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:28:54.479033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:28:54.479000 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:28:58.547545 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:28:58.547482 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:29:13.587738 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:29:13.587702 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:29:28.543135 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:29:28.543100 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:29:38.245480 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:29:38.245448 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:29:58.543471 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:29:58.543390 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.168734 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd"] Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169669 4933 cpu_manager.go:414] "containerMap: removing container" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169688 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169721 4933 cpu_manager.go:414] "containerMap: removing container" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169729 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169750 4933 cpu_manager.go:414] "containerMap: removing container" podUID="500f517d-042b-4a76-9c76-86e18a298464" containerName="revoke-keys" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169758 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500f517d-042b-4a76-9c76-86e18a298464" containerName="revoke-keys" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169799 4933 cpu_manager.go:414] "containerMap: removing container" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169805 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169825 4933 cpu_manager.go:414] "containerMap: removing container" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.169832 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.170000 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6faa2f-6fd1-43cd-8d36-d020bab4b638" containerName="main" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.170014 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ddd21c-c122-4046-9804-c3bcd2b34bc3" containerName="istio-proxy" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.170032 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="500f517d-042b-4a76-9c76-86e18a298464" containerName="revoke-keys" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.170051 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0ca645-0bf9-4e47-9cd3-222af475fff6" containerName="main" Oct 30 22:30:00.171139 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.170060 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf04c1d-4538-4574-aa8c-6b44e958e092" containerName="istio-proxy" Oct 30 22:30:00.175705 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.175676 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.264625 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.264594 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd"] Oct 30 22:30:00.265053 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.265010 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ba65a1-2561-4307-8106-a9db1d797f15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.265199 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.265092 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtt6x\" (UniqueName: \"kubernetes.io/projected/53ba65a1-2561-4307-8106-a9db1d797f15-kube-api-access-xtt6x\") pod \"maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.319176 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.319151 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:30:00.330946 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.330925 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:30:00.339766 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.339746 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d786g\"" Oct 30 22:30:00.365755 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.365735 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtt6x\" (UniqueName: \"kubernetes.io/projected/53ba65a1-2561-4307-8106-a9db1d797f15-kube-api-access-xtt6x\") pod \"maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.365855 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.365803 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ba65a1-2561-4307-8106-a9db1d797f15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.366405 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.366382 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ba65a1-2561-4307-8106-a9db1d797f15-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.425512 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.425429 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtt6x\" (UniqueName: \"kubernetes.io/projected/53ba65a1-2561-4307-8106-a9db1d797f15-kube-api-access-xtt6x\") pod \"maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.488652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.488632 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:00.859832 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:00.859810 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd"] Oct 30 22:30:00.861668 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:30:00.861637 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ba65a1_2561_4307_8106_a9db1d797f15.slice/crio-0813821b942d2a8aab1baecea3d5f2d9e544834e62ed956048108e3a80c01584 WatchSource:0}: Error finding container 0813821b942d2a8aab1baecea3d5f2d9e544834e62ed956048108e3a80c01584: Status 404 returned error can't find the container with id 0813821b942d2a8aab1baecea3d5f2d9e544834e62ed956048108e3a80c01584 Oct 30 22:30:01.829186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:01.829110 4933 generic.go:356] "Generic (PLEG): container finished" podID="53ba65a1-2561-4307-8106-a9db1d797f15" containerID="1be9c3d0882fd245e11ff1fb875885225ed4de35ec13025d614ca01b319a28c8" exitCode=60 Oct 30 22:30:01.829186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:01.829180 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" event={"ID":"53ba65a1-2561-4307-8106-a9db1d797f15","Type":"ContainerDied","Data":"1be9c3d0882fd245e11ff1fb875885225ed4de35ec13025d614ca01b319a28c8"} Oct 30 22:30:01.829601 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:01.829217 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" event={"ID":"53ba65a1-2561-4307-8106-a9db1d797f15","Type":"ContainerStarted","Data":"0813821b942d2a8aab1baecea3d5f2d9e544834e62ed956048108e3a80c01584"} Oct 30 22:30:01.829601 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:01.829401 4933 scope.go:122] "RemoveContainer" containerID="1be9c3d0882fd245e11ff1fb875885225ed4de35ec13025d614ca01b319a28c8" Oct 30 22:30:02.836461 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:02.836399 4933 generic.go:356] "Generic (PLEG): container finished" podID="53ba65a1-2561-4307-8106-a9db1d797f15" containerID="7607ea8ea8ca77d9cb00eae27f9a1367c039ce9f05051417dcc07d7fd2909516" exitCode=60 Oct 30 22:30:02.836461 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:02.836449 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" event={"ID":"53ba65a1-2561-4307-8106-a9db1d797f15","Type":"ContainerDied","Data":"7607ea8ea8ca77d9cb00eae27f9a1367c039ce9f05051417dcc07d7fd2909516"} Oct 30 22:30:02.836870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:02.836533 4933 scope.go:122] "RemoveContainer" containerID="1be9c3d0882fd245e11ff1fb875885225ed4de35ec13025d614ca01b319a28c8" Oct 30 22:30:02.836870 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:02.836683 4933 scope.go:122] "RemoveContainer" containerID="7607ea8ea8ca77d9cb00eae27f9a1367c039ce9f05051417dcc07d7fd2909516" Oct 30 22:30:02.836993 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:30:02.836903 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd_odh-ai-gateway-infra(53ba65a1-2561-4307-8106-a9db1d797f15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" Oct 30 22:30:03.843889 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:03.843860 4933 scope.go:122] "RemoveContainer" containerID="7607ea8ea8ca77d9cb00eae27f9a1367c039ce9f05051417dcc07d7fd2909516" Oct 30 22:30:03.844288 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:30:03.844033 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd_odh-ai-gateway-infra(53ba65a1-2561-4307-8106-a9db1d797f15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" Oct 30 22:30:09.367137 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:09.367104 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:30:13.459765 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:13.459736 4933 scope.go:122] "RemoveContainer" containerID="7607ea8ea8ca77d9cb00eae27f9a1367c039ce9f05051417dcc07d7fd2909516" Oct 30 22:30:13.906303 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:13.906274 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" event={"ID":"53ba65a1-2561-4307-8106-a9db1d797f15","Type":"ContainerStarted","Data":"4ee75a9bca8b1cf9db9dfdbce3491cffc8e03394335d4df6e5c316a37b631eb8"} Oct 30 22:30:14.023554 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:14.023458 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" podStartSLOduration=14.023445718 podStartE2EDuration="14.023445718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:14.022105658 +0000 UTC m=+2862.703255555" watchObservedRunningTime="2026-10-30 22:30:14.023445718 +0000 UTC m=+2862.704595611" Oct 30 22:30:14.914602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:14.914569 4933 generic.go:356] "Generic (PLEG): container finished" podID="53ba65a1-2561-4307-8106-a9db1d797f15" containerID="4ee75a9bca8b1cf9db9dfdbce3491cffc8e03394335d4df6e5c316a37b631eb8" exitCode=60 Oct 30 22:30:14.915078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:14.914642 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" event={"ID":"53ba65a1-2561-4307-8106-a9db1d797f15","Type":"ContainerDied","Data":"4ee75a9bca8b1cf9db9dfdbce3491cffc8e03394335d4df6e5c316a37b631eb8"} Oct 30 22:30:14.915078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:14.914721 4933 scope.go:122] "RemoveContainer" containerID="7607ea8ea8ca77d9cb00eae27f9a1367c039ce9f05051417dcc07d7fd2909516" Oct 30 22:30:14.915078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:14.915035 4933 scope.go:122] "RemoveContainer" containerID="4ee75a9bca8b1cf9db9dfdbce3491cffc8e03394335d4df6e5c316a37b631eb8" Oct 30 22:30:14.915287 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:30:14.915264 4933 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd_odh-ai-gateway-infra(53ba65a1-2561-4307-8106-a9db1d797f15)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" Oct 30 22:30:15.039518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:15.039481 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd"] Oct 30 22:30:16.064545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.064524 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:16.104276 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.104252 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ba65a1-2561-4307-8106-a9db1d797f15-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ba65a1-2561-4307-8106-a9db1d797f15-maas-api-service-ca\") pod \"53ba65a1-2561-4307-8106-a9db1d797f15\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " Oct 30 22:30:16.104424 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.104346 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ba65a1-2561-4307-8106-a9db1d797f15-kube-api-access-xtt6x\" (UniqueName: \"kubernetes.io/projected/53ba65a1-2561-4307-8106-a9db1d797f15-kube-api-access-xtt6x\") pod \"53ba65a1-2561-4307-8106-a9db1d797f15\" (UID: \"53ba65a1-2561-4307-8106-a9db1d797f15\") " Oct 30 22:30:16.104697 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.104662 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ba65a1-2561-4307-8106-a9db1d797f15-maas-api-service-ca" pod "53ba65a1-2561-4307-8106-a9db1d797f15" (UID: "53ba65a1-2561-4307-8106-a9db1d797f15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:16.106652 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.106629 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ba65a1-2561-4307-8106-a9db1d797f15-kube-api-access-xtt6x" pod "53ba65a1-2561-4307-8106-a9db1d797f15" (UID: "53ba65a1-2561-4307-8106-a9db1d797f15"). InnerVolumeSpecName "kube-api-access-xtt6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:16.205777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.205690 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53ba65a1-2561-4307-8106-a9db1d797f15-maas-api-service-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.205777 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.205716 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtt6x\" (UniqueName: \"kubernetes.io/projected/53ba65a1-2561-4307-8106-a9db1d797f15-kube-api-access-xtt6x\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:16.929305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.929269 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" event={"ID":"53ba65a1-2561-4307-8106-a9db1d797f15","Type":"ContainerDied","Data":"0813821b942d2a8aab1baecea3d5f2d9e544834e62ed956048108e3a80c01584"} Oct 30 22:30:16.929305 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.929299 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd" Oct 30 22:30:16.929616 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:16.929340 4933 scope.go:122] "RemoveContainer" containerID="4ee75a9bca8b1cf9db9dfdbce3491cffc8e03394335d4df6e5c316a37b631eb8" Oct 30 22:30:17.097178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:17.097150 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd"] Oct 30 22:30:17.147823 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:17.147798 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-9513bb-29889996z5zd"] Oct 30 22:30:17.465312 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:17.465285 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" path="/var/lib/kubelet/pods/53ba65a1-2561-4307-8106-a9db1d797f15/volumes" Oct 30 22:30:28.545976 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:30:28.545936 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:30:28.970996 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:28.970922 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:30:56.326905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.326874 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz"] Oct 30 22:30:56.327371 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.327080 4933 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" podUID="7db7387c-8320-4580-a561-121bd21ff2d9" containerName="manager" containerID="cri-o://2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520" gracePeriod=10 Oct 30 22:30:56.591115 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.591049 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 22:30:56.760122 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.760093 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db7387c-8320-4580-a561-121bd21ff2d9-kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/7db7387c-8320-4580-a561-121bd21ff2d9-kube-api-access-92c8f\") pod \"7db7387c-8320-4580-a561-121bd21ff2d9\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " Oct 30 22:30:56.760310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.760144 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db7387c-8320-4580-a561-121bd21ff2d9-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db7387c-8320-4580-a561-121bd21ff2d9-extensions-socket-volume\") pod \"7db7387c-8320-4580-a561-121bd21ff2d9\" (UID: \"7db7387c-8320-4580-a561-121bd21ff2d9\") " Oct 30 22:30:56.760886 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.760796 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db7387c-8320-4580-a561-121bd21ff2d9-extensions-socket-volume" pod "7db7387c-8320-4580-a561-121bd21ff2d9" (UID: "7db7387c-8320-4580-a561-121bd21ff2d9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:56.762378 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.762356 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db7387c-8320-4580-a561-121bd21ff2d9-kube-api-access-92c8f" pod "7db7387c-8320-4580-a561-121bd21ff2d9" (UID: "7db7387c-8320-4580-a561-121bd21ff2d9"). InnerVolumeSpecName "kube-api-access-92c8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:56.861896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.861822 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92c8f\" (UniqueName: \"kubernetes.io/projected/7db7387c-8320-4580-a561-121bd21ff2d9-kube-api-access-92c8f\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:56.861896 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:56.861851 4933 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7db7387c-8320-4580-a561-121bd21ff2d9-extensions-socket-volume\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:30:57.145518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.145422 4933 generic.go:356] "Generic (PLEG): container finished" podID="7db7387c-8320-4580-a561-121bd21ff2d9" containerID="2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520" exitCode=0 Oct 30 22:30:57.145518 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.145475 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" Oct 30 22:30:57.146541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.145476 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" event={"ID":"7db7387c-8320-4580-a561-121bd21ff2d9","Type":"ContainerDied","Data":"2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520"} Oct 30 22:30:57.146541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.146163 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz" event={"ID":"7db7387c-8320-4580-a561-121bd21ff2d9","Type":"ContainerDied","Data":"0123b693f037f665c6cb053d2060e40dcf13fbec36459cd14a6d9ea5e59dcc36"} Oct 30 22:30:57.146541 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.146199 4933 scope.go:122] "RemoveContainer" containerID="2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520" Oct 30 22:30:57.159740 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.159723 4933 scope.go:122] "RemoveContainer" containerID="2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520" Oct 30 22:30:57.160017 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:30:57.159999 4933 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520\": container with ID starting with 2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520 not found: ID does not exist" containerID="2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520" Oct 30 22:30:57.160092 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.160024 4933 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520"} err="failed to get container status \"2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520\": rpc error: code = NotFound desc = could not find container \"2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520\": container with ID starting with 2a2dd61e0bfebb21ca5cf19c4ffa24efadda34497caa10a598f602fd45b5b520 not found: ID does not exist" Oct 30 22:30:57.313073 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.313037 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz"] Oct 30 22:30:57.317049 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.317025 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w57qz"] Oct 30 22:30:57.464965 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:30:57.464891 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db7387c-8320-4580-a561-121bd21ff2d9" path="/var/lib/kubelet/pods/7db7387c-8320-4580-a561-121bd21ff2d9/volumes" Oct 30 22:30:58.542896 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:30:58.542833 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:31:28.542852 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:31:28.542779 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:31:58.542534 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:31:58.542473 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:31:59.872750 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.872720 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn"] Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873121 4933 cpu_manager.go:414] "containerMap: removing container" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873133 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873143 4933 cpu_manager.go:414] "containerMap: removing container" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873147 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873165 4933 cpu_manager.go:414] "containerMap: removing container" podUID="7db7387c-8320-4580-a561-121bd21ff2d9" containerName="manager" Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873170 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db7387c-8320-4580-a561-121bd21ff2d9" containerName="manager" Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873248 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:31:59.873251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873256 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:31:59.873586 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.873263 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db7387c-8320-4580-a561-121bd21ff2d9" containerName="manager" Oct 30 22:31:59.877011 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.876985 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:31:59.923375 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.923346 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn"] Oct 30 22:31:59.971786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.971759 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vp8s4\"" Oct 30 22:31:59.979702 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.979676 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rkx\" (UniqueName: \"kubernetes.io/projected/880c49b5-0c92-43df-8e8d-091d4d1e19cc-kube-api-access-s8rkx\") pod \"kuadrant-operator-controller-manager-678d87766f-6prnn\" (UID: \"880c49b5-0c92-43df-8e8d-091d4d1e19cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:31:59.979812 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:31:59.979752 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880c49b5-0c92-43df-8e8d-091d4d1e19cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6prnn\" (UID: \"880c49b5-0c92-43df-8e8d-091d4d1e19cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:00.080279 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:00.080252 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880c49b5-0c92-43df-8e8d-091d4d1e19cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6prnn\" (UID: \"880c49b5-0c92-43df-8e8d-091d4d1e19cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:00.080391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:00.080313 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rkx\" (UniqueName: \"kubernetes.io/projected/880c49b5-0c92-43df-8e8d-091d4d1e19cc-kube-api-access-s8rkx\") pod \"kuadrant-operator-controller-manager-678d87766f-6prnn\" (UID: \"880c49b5-0c92-43df-8e8d-091d4d1e19cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:00.080665 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:00.080646 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/880c49b5-0c92-43df-8e8d-091d4d1e19cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6prnn\" (UID: \"880c49b5-0c92-43df-8e8d-091d4d1e19cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:00.139620 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:00.139562 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rkx\" (UniqueName: \"kubernetes.io/projected/880c49b5-0c92-43df-8e8d-091d4d1e19cc-kube-api-access-s8rkx\") pod \"kuadrant-operator-controller-manager-678d87766f-6prnn\" (UID: \"880c49b5-0c92-43df-8e8d-091d4d1e19cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:00.191807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:00.191782 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:00.663403 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:00.663368 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn"] Oct 30 22:32:00.701031 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:32:00.701001 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880c49b5_0c92_43df_8e8d_091d4d1e19cc.slice/crio-2279ff84d5879f46dfd947f7e97bfaf7d7f5a71ac7f3c4132f4270b53b658b12 WatchSource:0}: Error finding container 2279ff84d5879f46dfd947f7e97bfaf7d7f5a71ac7f3c4132f4270b53b658b12: Status 404 returned error can't find the container with id 2279ff84d5879f46dfd947f7e97bfaf7d7f5a71ac7f3c4132f4270b53b658b12 Oct 30 22:32:01.481002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:01.480970 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" event={"ID":"880c49b5-0c92-43df-8e8d-091d4d1e19cc","Type":"ContainerStarted","Data":"849d4cfcf9c23a1b1d2707165cc88a8d5fb8834d0f2587948071a07e33ef0fce"} Oct 30 22:32:01.481456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:01.481021 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" event={"ID":"880c49b5-0c92-43df-8e8d-091d4d1e19cc","Type":"ContainerStarted","Data":"2279ff84d5879f46dfd947f7e97bfaf7d7f5a71ac7f3c4132f4270b53b658b12"} Oct 30 22:32:01.481456 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:01.481041 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:01.702398 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:01.702352 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" podStartSLOduration=2.7023384999999998 podStartE2EDuration="2.7023385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:01.650056641 +0000 UTC m=+2970.331206558" watchObservedRunningTime="2026-10-30 22:32:01.7023385 +0000 UTC m=+2970.383488392" Oct 30 22:32:12.488789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:12.488759 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6prnn" Oct 30 22:32:20.849000 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:20.843757 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:32:28.544262 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:32:28.544225 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:32:32.022257 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:32.022228 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:32:32.029485 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:32.029466 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:32:38.044404 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:38.044368 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:32:45.905348 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:32:45.905267 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:32:58.543636 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:32:58.543596 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:33:06.978882 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:33:06.978850 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:33:25.408391 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:33:25.408359 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:33:28.544284 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:33:28.544239 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:33:43.579450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:33:43.579411 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:33:58.542061 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:33:58.542013 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:34:11.669058 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:34:11.668971 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:34:28.544561 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:34:28.544523 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:34:37.482408 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:34:37.482374 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:34:58.543973 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:34:58.543934 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:34:59.870436 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:34:59.870402 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:35:20.352432 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:35:20.352395 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:35:22.829733 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:35:22.829700 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:35:22.830401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:35:22.830367 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq" podUID="acef2b25-5c28-44db-8c2b-ba32a12ddce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:35:28.542702 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:35:28.542664 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:35:58.543653 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:35:58.543576 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:36:02.687880 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:36:02.687842 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:36:28.544377 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:36:28.544342 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:36:29.167099 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:36:29.167065 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:36:31.861516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:36:31.860618 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:36:31.861516 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:36:31.860697 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:36:44.634019 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:36:44.633980 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:36:56.349186 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:36:56.349146 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:36:58.543726 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:36:58.543689 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:37:08.046185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:37:08.046151 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:37:28.542772 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:37:28.542682 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:37:31.546447 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:37:31.546415 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:37:32.059958 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:37:32.059931 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:37:32.070084 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:37:32.070060 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:37:55.350877 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:37:55.350835 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:37:58.544463 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:37:58.544428 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:38:19.367528 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.367480 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml"] Oct 30 22:38:19.367909 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.367897 4933 cpu_manager.go:414] "containerMap: removing container" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:38:19.367971 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.367913 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:38:19.368030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.368020 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ba65a1-2561-4307-8106-a9db1d797f15" containerName="cleanup" Oct 30 22:38:19.371131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.371116 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.426263 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.426240 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-69npn\"" Oct 30 22:38:19.477133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.477108 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:38:19.477133 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.477114 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:38:19.479470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.479450 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml"] Oct 30 22:38:19.515466 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.515440 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.515628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.515482 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.515628 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.515600 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdtz\" (UniqueName: \"kubernetes.io/projected/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-kube-api-access-zkdtz\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.524821 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.524800 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:38:19.616018 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.615988 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdtz\" (UniqueName: \"kubernetes.io/projected/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-kube-api-access-zkdtz\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.616185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.616065 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.616185 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.616119 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.616807 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.616781 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.618715 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.618666 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.674071 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.674048 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdtz\" (UniqueName: \"kubernetes.io/projected/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-kube-api-access-zkdtz\") pod \"maas-api-revoke-keys-e2e-ait-f56a3efc-85kml\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:19.684173 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:19.684148 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:20.058770 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:20.058746 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml"] Oct 30 22:38:20.059865 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:38:20.059839 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe27c4d_94b1_40bc_8bd5_1bf987dce83e.slice/crio-204eb4b9809201e3021bcda43202bac539be962eac3965945f493afe9894eff1 WatchSource:0}: Error finding container 204eb4b9809201e3021bcda43202bac539be962eac3965945f493afe9894eff1: Status 404 returned error can't find the container with id 204eb4b9809201e3021bcda43202bac539be962eac3965945f493afe9894eff1 Oct 30 22:38:20.643326 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:20.643294 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" event={"ID":"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e","Type":"ContainerStarted","Data":"ec96133cf93bb3031a32d2c00472f0f2658f6c3205e66e0a8a2b11d01ebb9681"} Oct 30 22:38:20.643725 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:20.643341 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" event={"ID":"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e","Type":"ContainerStarted","Data":"204eb4b9809201e3021bcda43202bac539be962eac3965945f493afe9894eff1"} Oct 30 22:38:20.761097 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:20.761051 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" podStartSLOduration=1.761037913 podStartE2EDuration="1.761037913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:20.759410544 +0000 UTC m=+3349.440560438" watchObservedRunningTime="2026-10-30 22:38:20.761037913 +0000 UTC m=+3349.442187810" Oct 30 22:38:21.649532 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:21.649484 4933 generic.go:356] "Generic (PLEG): container finished" podID="bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" containerID="ec96133cf93bb3031a32d2c00472f0f2658f6c3205e66e0a8a2b11d01ebb9681" exitCode=0 Oct 30 22:38:21.649981 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:21.649594 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" event={"ID":"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e","Type":"ContainerDied","Data":"ec96133cf93bb3031a32d2c00472f0f2658f6c3205e66e0a8a2b11d01ebb9681"} Oct 30 22:38:22.788389 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:22.788366 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:22.945749 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:22.945680 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-openshift-ca\") pod \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " Oct 30 22:38:22.945749 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:22.945733 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-xks-ca\") pod \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " Oct 30 22:38:22.945923 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:22.945838 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-kube-api-access-zkdtz\" (UniqueName: \"kubernetes.io/projected/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-kube-api-access-zkdtz\") pod \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\" (UID: \"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e\") " Oct 30 22:38:22.946033 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:22.946012 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-openshift-ca" pod "bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" (UID: "bfe27c4d-94b1-40bc-8bd5-1bf987dce83e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:22.947959 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:22.947930 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-xks-ca" pod "bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" (UID: "bfe27c4d-94b1-40bc-8bd5-1bf987dce83e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:22.948129 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:22.948107 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-kube-api-access-zkdtz" pod "bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" (UID: "bfe27c4d-94b1-40bc-8bd5-1bf987dce83e"). InnerVolumeSpecName "kube-api-access-zkdtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:23.046933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:23.046912 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:23.046933 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:23.046934 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:23.047074 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:23.046944 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkdtz\" (UniqueName: \"kubernetes.io/projected/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e-kube-api-access-zkdtz\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:38:23.662510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:23.662473 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" Oct 30 22:38:23.662510 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:23.662476 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml" event={"ID":"bfe27c4d-94b1-40bc-8bd5-1bf987dce83e","Type":"ContainerDied","Data":"204eb4b9809201e3021bcda43202bac539be962eac3965945f493afe9894eff1"} Oct 30 22:38:23.662717 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:23.662538 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="204eb4b9809201e3021bcda43202bac539be962eac3965945f493afe9894eff1" Oct 30 22:38:28.542848 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:38:28.542807 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:38:38.475230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:38.475201 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml"] Oct 30 22:38:38.525362 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:38.525333 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f56a3efc-85kml"] Oct 30 22:38:39.464057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:39.464025 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" path="/var/lib/kubelet/pods/bfe27c4d-94b1-40bc-8bd5-1bf987dce83e/volumes" Oct 30 22:38:40.952824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:38:40.952719 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:38:58.542665 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:38:58.542629 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:39:01.066085 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:39:01.060325 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:39:28.543275 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:39:28.543240 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:39:58.542916 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:39:58.542879 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:40:28.544651 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:40:28.544573 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:40:30.891607 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.891579 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc"] Oct 30 22:40:30.891977 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.891950 4933 cpu_manager.go:414] "containerMap: removing container" podUID="bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" containerName="revoke-keys" Oct 30 22:40:30.891977 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.891960 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" containerName="revoke-keys" Oct 30 22:40:30.892054 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.892043 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe27c4d-94b1-40bc-8bd5-1bf987dce83e" containerName="revoke-keys" Oct 30 22:40:30.895128 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.895112 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:30.951178 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.951157 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:40:30.999347 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.999322 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:40:30.999473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.999380 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:40:30.999473 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:30.999414 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-69npn\"" Oct 30 22:40:31.001935 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.001911 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l94j\" (UniqueName: \"kubernetes.io/projected/2f709912-2608-4511-a1dd-1d940a2f3494-kube-api-access-9l94j\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.002056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.001938 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc"] Oct 30 22:40:31.002056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.002011 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.002145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.002072 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.103246 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.103223 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.103370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.103265 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.103370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.103328 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l94j\" (UniqueName: \"kubernetes.io/projected/2f709912-2608-4511-a1dd-1d940a2f3494-kube-api-access-9l94j\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.103857 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.103838 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.105780 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.105758 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.162194 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.162144 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l94j\" (UniqueName: \"kubernetes.io/projected/2f709912-2608-4511-a1dd-1d940a2f3494-kube-api-access-9l94j\") pod \"maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.208105 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.208078 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:31.586860 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:31.586834 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc"] Oct 30 22:40:31.588480 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:40:31.588454 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f709912_2608_4511_a1dd_1d940a2f3494.slice/crio-9ada230e6a4cd97874bdc97182c11e507277bbafaeb2f0e400c08d0b149fdc29 WatchSource:0}: Error finding container 9ada230e6a4cd97874bdc97182c11e507277bbafaeb2f0e400c08d0b149fdc29: Status 404 returned error can't find the container with id 9ada230e6a4cd97874bdc97182c11e507277bbafaeb2f0e400c08d0b149fdc29 Oct 30 22:40:32.330936 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:32.330898 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" event={"ID":"2f709912-2608-4511-a1dd-1d940a2f3494","Type":"ContainerStarted","Data":"94dc07da7333178034a3002e87ea624a0648f61a06a4210e3302fee281106a6d"} Oct 30 22:40:32.331399 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:32.330947 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" event={"ID":"2f709912-2608-4511-a1dd-1d940a2f3494","Type":"ContainerStarted","Data":"9ada230e6a4cd97874bdc97182c11e507277bbafaeb2f0e400c08d0b149fdc29"} Oct 30 22:40:32.449540 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:32.449471 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" podStartSLOduration=2.449458395 podStartE2EDuration="2.449458395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:32.449104304 +0000 UTC m=+3481.130254203" watchObservedRunningTime="2026-10-30 22:40:32.449458395 +0000 UTC m=+3481.130608288" Oct 30 22:40:33.337789 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:33.337756 4933 generic.go:356] "Generic (PLEG): container finished" podID="2f709912-2608-4511-a1dd-1d940a2f3494" containerID="94dc07da7333178034a3002e87ea624a0648f61a06a4210e3302fee281106a6d" exitCode=0 Oct 30 22:40:33.338224 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:33.337839 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" event={"ID":"2f709912-2608-4511-a1dd-1d940a2f3494","Type":"ContainerDied","Data":"94dc07da7333178034a3002e87ea624a0648f61a06a4210e3302fee281106a6d"} Oct 30 22:40:34.469730 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.469709 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:34.633209 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.633141 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-openshift-ca\") pod \"2f709912-2608-4511-a1dd-1d940a2f3494\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " Oct 30 22:40:34.633355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.633234 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f709912-2608-4511-a1dd-1d940a2f3494-kube-api-access-9l94j\" (UniqueName: \"kubernetes.io/projected/2f709912-2608-4511-a1dd-1d940a2f3494-kube-api-access-9l94j\") pod \"2f709912-2608-4511-a1dd-1d940a2f3494\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " Oct 30 22:40:34.633355 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.633271 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-xks-ca\") pod \"2f709912-2608-4511-a1dd-1d940a2f3494\" (UID: \"2f709912-2608-4511-a1dd-1d940a2f3494\") " Oct 30 22:40:34.633589 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.633563 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-openshift-ca" pod "2f709912-2608-4511-a1dd-1d940a2f3494" (UID: "2f709912-2608-4511-a1dd-1d940a2f3494"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:40:34.633824 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.633802 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:34.635600 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.635576 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-xks-ca" pod "2f709912-2608-4511-a1dd-1d940a2f3494" (UID: "2f709912-2608-4511-a1dd-1d940a2f3494"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:40:34.635692 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.635633 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f709912-2608-4511-a1dd-1d940a2f3494-kube-api-access-9l94j" pod "2f709912-2608-4511-a1dd-1d940a2f3494" (UID: "2f709912-2608-4511-a1dd-1d940a2f3494"). InnerVolumeSpecName "kube-api-access-9l94j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:40:34.734892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.734857 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l94j\" (UniqueName: \"kubernetes.io/projected/2f709912-2608-4511-a1dd-1d940a2f3494-kube-api-access-9l94j\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:34.734892 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:34.734889 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f709912-2608-4511-a1dd-1d940a2f3494-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:40:35.350602 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:35.350569 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" Oct 30 22:40:35.350781 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:35.350606 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc" event={"ID":"2f709912-2608-4511-a1dd-1d940a2f3494","Type":"ContainerDied","Data":"9ada230e6a4cd97874bdc97182c11e507277bbafaeb2f0e400c08d0b149fdc29"} Oct 30 22:40:35.350781 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:35.350654 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ada230e6a4cd97874bdc97182c11e507277bbafaeb2f0e400c08d0b149fdc29" Oct 30 22:40:57.566615 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:57.566589 4933 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc"] Oct 30 22:40:57.568470 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:57.568450 4933 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-29f146-ph2lc"] Oct 30 22:40:58.543246 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:40:58.543210 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:40:59.464575 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:40:59.464540 4933 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f709912-2608-4511-a1dd-1d940a2f3494" path="/var/lib/kubelet/pods/2f709912-2608-4511-a1dd-1d940a2f3494/volumes" Oct 30 22:41:00.597396 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:00.597366 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:41:08.884386 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:08.884351 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:41:23.260160 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:23.260126 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:41:28.546308 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:41:28.546265 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:41:31.331143 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.331111 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68"] Oct 30 22:41:31.331656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.331554 4933 cpu_manager.go:414] "containerMap: removing container" podUID="2f709912-2608-4511-a1dd-1d940a2f3494" containerName="revoke-keys" Oct 30 22:41:31.331656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.331566 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f709912-2608-4511-a1dd-1d940a2f3494" containerName="revoke-keys" Oct 30 22:41:31.331656 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.331631 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f709912-2608-4511-a1dd-1d940a2f3494" containerName="revoke-keys" Oct 30 22:41:31.334873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.334858 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.405707 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.405671 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjqc\" (UniqueName: \"kubernetes.io/projected/134c7d44-e7a3-4463-9436-1c93da07e6b5-kube-api-access-4hjqc\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.405842 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.405735 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.405893 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.405867 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.439759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.439736 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:41:31.439759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.439738 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:41:31.439930 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.439782 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:41:31.439930 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.439798 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-69npn\"" Oct 30 22:41:31.441968 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.441948 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68"] Oct 30 22:41:31.506614 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.506589 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.506788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.506651 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjqc\" (UniqueName: \"kubernetes.io/projected/134c7d44-e7a3-4463-9436-1c93da07e6b5-kube-api-access-4hjqc\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.506788 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.506704 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.509039 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.509016 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.558751 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.558730 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:41:31.567869 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.567849 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.614105 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.614057 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:41:31.625131 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.625112 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjqc\" (UniqueName: \"kubernetes.io/projected/134c7d44-e7a3-4463-9436-1c93da07e6b5-kube-api-access-4hjqc\") pod \"maas-api-revoke-sub-f03629db-glz68\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:31.648069 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:31.648048 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:32.021284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:32.021248 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68"] Oct 30 22:41:32.022542 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:41:32.022512 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134c7d44_e7a3_4463_9436_1c93da07e6b5.slice/crio-9d3438cf5a000cfef8cb8a7314c2414416ecbe7fead10f66517e4556b9ed7c4a WatchSource:0}: Error finding container 9d3438cf5a000cfef8cb8a7314c2414416ecbe7fead10f66517e4556b9ed7c4a: Status 404 returned error can't find the container with id 9d3438cf5a000cfef8cb8a7314c2414416ecbe7fead10f66517e4556b9ed7c4a Oct 30 22:41:32.741067 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:32.741039 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" event={"ID":"134c7d44-e7a3-4463-9436-1c93da07e6b5","Type":"ContainerStarted","Data":"7fe870bfbbbeb2f86b30600f5515d55a43a60bcff34b90047f5685575071470e"} Oct 30 22:41:32.741465 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:32.741086 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" event={"ID":"134c7d44-e7a3-4463-9436-1c93da07e6b5","Type":"ContainerStarted","Data":"9d3438cf5a000cfef8cb8a7314c2414416ecbe7fead10f66517e4556b9ed7c4a"} Oct 30 22:41:32.861626 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:32.861582 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" podStartSLOduration=1.861567712 podStartE2EDuration="1.861567712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:32.859825422 +0000 UTC m=+3541.540975315" watchObservedRunningTime="2026-10-30 22:41:32.861567712 +0000 UTC m=+3541.542717599" Oct 30 22:41:33.747030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:33.746997 4933 generic.go:356] "Generic (PLEG): container finished" podID="134c7d44-e7a3-4463-9436-1c93da07e6b5" containerID="7fe870bfbbbeb2f86b30600f5515d55a43a60bcff34b90047f5685575071470e" exitCode=0 Oct 30 22:41:33.747445 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:33.747086 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" event={"ID":"134c7d44-e7a3-4463-9436-1c93da07e6b5","Type":"ContainerDied","Data":"7fe870bfbbbeb2f86b30600f5515d55a43a60bcff34b90047f5685575071470e"} Oct 30 22:41:34.891401 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:34.891374 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:35.036584 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.036559 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134c7d44-e7a3-4463-9436-1c93da07e6b5-kube-api-access-4hjqc\" (UniqueName: \"kubernetes.io/projected/134c7d44-e7a3-4463-9436-1c93da07e6b5-kube-api-access-4hjqc\") pod \"134c7d44-e7a3-4463-9436-1c93da07e6b5\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " Oct 30 22:41:35.036744 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.036660 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-openshift-ca\") pod \"134c7d44-e7a3-4463-9436-1c93da07e6b5\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " Oct 30 22:41:35.036744 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.036694 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-xks-ca\") pod \"134c7d44-e7a3-4463-9436-1c93da07e6b5\" (UID: \"134c7d44-e7a3-4463-9436-1c93da07e6b5\") " Oct 30 22:41:35.037051 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.037028 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-openshift-ca" pod "134c7d44-e7a3-4463-9436-1c93da07e6b5" (UID: "134c7d44-e7a3-4463-9436-1c93da07e6b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:35.038979 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.038944 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-xks-ca" pod "134c7d44-e7a3-4463-9436-1c93da07e6b5" (UID: "134c7d44-e7a3-4463-9436-1c93da07e6b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:35.039100 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.039034 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134c7d44-e7a3-4463-9436-1c93da07e6b5-kube-api-access-4hjqc" pod "134c7d44-e7a3-4463-9436-1c93da07e6b5" (UID: "134c7d44-e7a3-4463-9436-1c93da07e6b5"). InnerVolumeSpecName "kube-api-access-4hjqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:35.137819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.137794 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjqc\" (UniqueName: \"kubernetes.io/projected/134c7d44-e7a3-4463-9436-1c93da07e6b5-kube-api-access-4hjqc\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:35.137819 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.137816 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:35.137945 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.137826 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134c7d44-e7a3-4463-9436-1c93da07e6b5-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:35.765759 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.765687 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" Oct 30 22:41:35.765927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.765686 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-glz68" event={"ID":"134c7d44-e7a3-4463-9436-1c93da07e6b5","Type":"ContainerDied","Data":"9d3438cf5a000cfef8cb8a7314c2414416ecbe7fead10f66517e4556b9ed7c4a"} Oct 30 22:41:35.765927 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:35.765800 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d3438cf5a000cfef8cb8a7314c2414416ecbe7fead10f66517e4556b9ed7c4a" Oct 30 22:41:44.265381 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:44.265342 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:41:50.492925 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.492887 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk"] Oct 30 22:41:50.494156 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.494129 4933 cpu_manager.go:414] "containerMap: removing container" podUID="134c7d44-e7a3-4463-9436-1c93da07e6b5" containerName="revoke-keys" Oct 30 22:41:50.494304 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.494287 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134c7d44-e7a3-4463-9436-1c93da07e6b5" containerName="revoke-keys" Oct 30 22:41:50.494579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.494561 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="134c7d44-e7a3-4463-9436-1c93da07e6b5" containerName="revoke-keys" Oct 30 22:41:50.501774 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.501753 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.616119 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.602192 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk"] Oct 30 22:41:50.616119 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.609390 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:41:50.616119 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.609535 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:41:50.616119 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.609623 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:41:50.646676 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.646650 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-69npn\"" Oct 30 22:41:50.702515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.702021 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mh4\" (UniqueName: \"kubernetes.io/projected/3de3909e-eda5-4e92-8bad-16b046ef465e-kube-api-access-c9mh4\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.702515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.702142 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.702515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.702196 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.808515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.807056 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.808515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.807125 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.808515 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.807181 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mh4\" (UniqueName: \"kubernetes.io/projected/3de3909e-eda5-4e92-8bad-16b046ef465e-kube-api-access-c9mh4\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.808772 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.808610 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.811037 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.811012 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:50.872798 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:50.872771 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mh4\" (UniqueName: \"kubernetes.io/projected/3de3909e-eda5-4e92-8bad-16b046ef465e-kube-api-access-c9mh4\") pod \"maas-api-revoke-sub-0165d7d2-dnmfk\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:51.118047 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:51.117956 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:51.655903 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:51.654120 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk"] Oct 30 22:41:51.901567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:51.901536 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" event={"ID":"3de3909e-eda5-4e92-8bad-16b046ef465e","Type":"ContainerStarted","Data":"da6806f99c83047405a16fb52923685794f5c6a77f99bf7029686c3ffc192e65"} Oct 30 22:41:52.908567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:52.908469 4933 generic.go:356] "Generic (PLEG): container finished" podID="3de3909e-eda5-4e92-8bad-16b046ef465e" containerID="c59967af9c59f9a6f6e8ad18c6f36b02f67c6fce58db6272556e12c019665a26" exitCode=0 Oct 30 22:41:52.908567 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:52.908521 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" event={"ID":"3de3909e-eda5-4e92-8bad-16b046ef465e","Type":"ContainerDied","Data":"c59967af9c59f9a6f6e8ad18c6f36b02f67c6fce58db6272556e12c019665a26"} Oct 30 22:41:54.050358 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.050329 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:54.150397 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.150373 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de3909e-eda5-4e92-8bad-16b046ef465e-kube-api-access-c9mh4\" (UniqueName: \"kubernetes.io/projected/3de3909e-eda5-4e92-8bad-16b046ef465e-kube-api-access-c9mh4\") pod \"3de3909e-eda5-4e92-8bad-16b046ef465e\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " Oct 30 22:41:54.150545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.150416 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-openshift-ca\") pod \"3de3909e-eda5-4e92-8bad-16b046ef465e\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " Oct 30 22:41:54.150545 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.150439 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-xks-ca\") pod \"3de3909e-eda5-4e92-8bad-16b046ef465e\" (UID: \"3de3909e-eda5-4e92-8bad-16b046ef465e\") " Oct 30 22:41:54.150796 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.150778 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-openshift-ca" pod "3de3909e-eda5-4e92-8bad-16b046ef465e" (UID: "3de3909e-eda5-4e92-8bad-16b046ef465e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:41:54.152647 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.152625 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-xks-ca" pod "3de3909e-eda5-4e92-8bad-16b046ef465e" (UID: "3de3909e-eda5-4e92-8bad-16b046ef465e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:54.152805 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.152785 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de3909e-eda5-4e92-8bad-16b046ef465e-kube-api-access-c9mh4" pod "3de3909e-eda5-4e92-8bad-16b046ef465e" (UID: "3de3909e-eda5-4e92-8bad-16b046ef465e"). InnerVolumeSpecName "kube-api-access-c9mh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:54.251375 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.251317 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mh4\" (UniqueName: \"kubernetes.io/projected/3de3909e-eda5-4e92-8bad-16b046ef465e-kube-api-access-c9mh4\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:54.251375 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.251340 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:54.251375 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.251350 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3de3909e-eda5-4e92-8bad-16b046ef465e-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:41:54.919612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.919578 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" event={"ID":"3de3909e-eda5-4e92-8bad-16b046ef465e","Type":"ContainerDied","Data":"da6806f99c83047405a16fb52923685794f5c6a77f99bf7029686c3ffc192e65"} Oct 30 22:41:54.919794 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.919639 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6806f99c83047405a16fb52923685794f5c6a77f99bf7029686c3ffc192e65" Oct 30 22:41:54.919794 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:41:54.919592 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dnmfk" Oct 30 22:41:58.542538 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:41:58.542503 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:42:02.282098 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.282061 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp"] Oct 30 22:42:02.282696 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.282678 4933 cpu_manager.go:414] "containerMap: removing container" podUID="3de3909e-eda5-4e92-8bad-16b046ef465e" containerName="revoke-keys" Oct 30 22:42:02.282756 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.282699 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de3909e-eda5-4e92-8bad-16b046ef465e" containerName="revoke-keys" Oct 30 22:42:02.282794 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.282776 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de3909e-eda5-4e92-8bad-16b046ef465e" containerName="revoke-keys" Oct 30 22:42:02.285786 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.285770 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.320420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.320388 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.320420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.320422 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.320659 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.320573 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8gh\" (UniqueName: \"kubernetes.io/projected/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-kube-api-access-hv8gh\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.389579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.389544 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:42:02.389579 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.389563 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:42:02.389816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.389613 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-69npn\"" Oct 30 22:42:02.389816 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.389606 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:42:02.391847 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.391829 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp"] Oct 30 22:42:02.422002 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.421979 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.422119 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.422007 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.422165 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.422137 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8gh\" (UniqueName: \"kubernetes.io/projected/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-kube-api-access-hv8gh\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.422809 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.422744 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.424375 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.424355 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.480028 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.479998 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8gh\" (UniqueName: \"kubernetes.io/projected/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-kube-api-access-hv8gh\") pod \"maas-api-revoke-sub-6b19338f-5tqnp\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.599833 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.599772 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:02.974783 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:02.974755 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp"] Oct 30 22:42:02.975921 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:42:02.975880 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad82aae6_0e74_43d0_afc9_0232c6c69ff2.slice/crio-e49a34fd3390844493bfb018ce91340a2bcf6efe59540e362bdab45c949d1b57 WatchSource:0}: Error finding container e49a34fd3390844493bfb018ce91340a2bcf6efe59540e362bdab45c949d1b57: Status 404 returned error can't find the container with id e49a34fd3390844493bfb018ce91340a2bcf6efe59540e362bdab45c949d1b57 Oct 30 22:42:03.973272 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:03.973234 4933 generic.go:356] "Generic (PLEG): container finished" podID="ad82aae6-0e74-43d0-afc9-0232c6c69ff2" containerID="0549ed00e4010de0fc8a8c0211e688e6f8efa375fddef38b11e3619ed34008c0" exitCode=0 Oct 30 22:42:03.974006 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:03.973981 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" event={"ID":"ad82aae6-0e74-43d0-afc9-0232c6c69ff2","Type":"ContainerDied","Data":"0549ed00e4010de0fc8a8c0211e688e6f8efa375fddef38b11e3619ed34008c0"} Oct 30 22:42:03.974107 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:03.974039 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" event={"ID":"ad82aae6-0e74-43d0-afc9-0232c6c69ff2","Type":"ContainerStarted","Data":"e49a34fd3390844493bfb018ce91340a2bcf6efe59540e362bdab45c949d1b57"} Oct 30 22:42:05.115966 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.115943 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:05.147260 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.147239 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-xks-ca\") pod \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " Oct 30 22:42:05.147400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.147283 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-kube-api-access-hv8gh\" (UniqueName: \"kubernetes.io/projected/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-kube-api-access-hv8gh\") pod \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " Oct 30 22:42:05.147400 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.147329 4933 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-openshift-ca\") pod \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\" (UID: \"ad82aae6-0e74-43d0-afc9-0232c6c69ff2\") " Oct 30 22:42:05.147753 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.147722 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-openshift-ca" pod "ad82aae6-0e74-43d0-afc9-0232c6c69ff2" (UID: "ad82aae6-0e74-43d0-afc9-0232c6c69ff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:42:05.149690 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.149667 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-kube-api-access-hv8gh" pod "ad82aae6-0e74-43d0-afc9-0232c6c69ff2" (UID: "ad82aae6-0e74-43d0-afc9-0232c6c69ff2"). InnerVolumeSpecName "kube-api-access-hv8gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:42:05.149801 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.149780 4933 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-xks-ca" pod "ad82aae6-0e74-43d0-afc9-0232c6c69ff2" (UID: "ad82aae6-0e74-43d0-afc9-0232c6c69ff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:42:05.248783 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.248715 4933 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv8gh\" (UniqueName: \"kubernetes.io/projected/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-kube-api-access-hv8gh\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:05.248783 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.248738 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-openshift-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:05.248783 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.248749 4933 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad82aae6-0e74-43d0-afc9-0232c6c69ff2-maas-api-xks-ca\") on node \"ip-10-0-131-250.us-west-1.compute.internal\" DevicePath \"\"" Oct 30 22:42:05.986009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.985975 4933 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" Oct 30 22:42:05.986009 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.985994 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5tqnp" event={"ID":"ad82aae6-0e74-43d0-afc9-0232c6c69ff2","Type":"ContainerDied","Data":"e49a34fd3390844493bfb018ce91340a2bcf6efe59540e362bdab45c949d1b57"} Oct 30 22:42:05.986213 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:05.986045 4933 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49a34fd3390844493bfb018ce91340a2bcf6efe59540e362bdab45c949d1b57" Oct 30 22:42:14.386054 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:14.386025 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-676b586c44-xvzr8_91184aa4-0bba-49fb-a9a7-c3fca31ce6e7/authorino/0.log" Oct 30 22:42:15.064230 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:15.064201 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h5bzd"] Oct 30 22:42:16.863643 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:16.863606 4933 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4s928 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:42:16.864450 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:16.864415 4933 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4s928" podUID="3a5021c0-a209-41c7-8416-f72d83edc4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:42:25.017081 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:25.017044 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-86z25_4bb1bfed-1923-4fb4-a7a9-7c34ae56655a/manager/0.log" Oct 30 22:42:25.768566 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:25.768533 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ntncq_c4a51477-e289-421a-8a6e-3d720975fdc2/manager/0.log" Oct 30 22:42:26.725320 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:26.725295 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jpwxz_387430b1-4ca6-4b40-80a4-6e64323f2f40/server/0.log" Oct 30 22:42:28.453196 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:28.453171 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dnmfk_3de3909e-eda5-4e92-8bad-16b046ef465e/revoke-keys/0.log" Oct 30 22:42:28.542820 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:42:28.542785 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:42:28.668666 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:28.668640 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-5tqnp_ad82aae6-0e74-43d0-afc9-0232c6c69ff2/revoke-keys/0.log" Oct 30 22:42:28.869282 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:28.869258 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-glz68_134c7d44-e7a3-4463-9436-1c93da07e6b5/revoke-keys/0.log" Oct 30 22:42:30.488421 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:30.488388 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-676b586c44-xvzr8_91184aa4-0bba-49fb-a9a7-c3fca31ce6e7/authorino/0.log" Oct 30 22:42:30.936757 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:30.936724 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zj45z_00f89aea-4186-454a-a88b-91ac4a55b503/manager/0.log" Oct 30 22:42:31.155874 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:31.155850 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tbpwm_c1b48785-35d1-41cf-b032-b74d4d74bfee/manager/0.log" Oct 30 22:42:31.620358 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:31.620331 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6prnn_880c49b5-0c92-43df-8e8d-091d4d1e19cc/manager/0.log" Oct 30 22:42:31.920556 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:31.920467 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h5bzd_4d577a18-2871-4312-97f6-8b1e7ecd1fb8/limitador/0.log" Oct 30 22:42:32.097955 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:32.097929 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:42:32.109154 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:32.109135 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-q69rp_d0ee0cf5-2de4-493c-85fe-f97c527d5635/console-operator/2.log" Oct 30 22:42:32.146709 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:32.146683 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zgqr7_2a86402f-2067-40c8-a53f-fe523aaebebc/manager/0.log" Oct 30 22:42:32.927749 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:32.927721 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d97d4qh_8ec5f880-07fb-4331-ad83-e193ebbf62f2/istio-proxy/0.log" Oct 30 22:42:33.809102 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:33.809075 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-2877x_fec7a677-1d2b-416f-8de9-0a65229fd10d/istio-proxy/0.log" Oct 30 22:42:34.272980 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:34.272907 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4s928_3a5021c0-a209-41c7-8416-f72d83edc4a1/istio-proxy/0.log" Oct 30 22:42:34.764813 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:34.764769 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq_acef2b25-5c28-44db-8c2b-ba32a12ddce6/istio-proxy/1.log" Oct 30 22:42:36.220310 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:36.220281 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k_e7a612dc-b817-4e16-90c3-a73530490ad9/istio-proxy/0.log" Oct 30 22:42:36.902370 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:36.902346 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7444c59677-nrb6q_3be119ed-6228-46e8-a71b-8f7fb6e812f8/router/0.log" Oct 30 22:42:37.447732 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:37.447700 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq_acef2b25-5c28-44db-8c2b-ba32a12ddce6/istio-proxy/0.log" Oct 30 22:42:40.661570 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:40.661538 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-676b586c44-xvzr8_91184aa4-0bba-49fb-a9a7-c3fca31ce6e7/authorino/0.log" Oct 30 22:42:50.070035 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:50.070004 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-86z25_4bb1bfed-1923-4fb4-a7a9-7c34ae56655a/manager/0.log" Oct 30 22:42:50.657274 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:50.657248 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-ntncq_c4a51477-e289-421a-8a6e-3d720975fdc2/manager/0.log" Oct 30 22:42:51.364873 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:51.364838 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jpwxz_387430b1-4ca6-4b40-80a4-6e64323f2f40/server/0.log" Oct 30 22:42:52.921646 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:52.921621 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dnmfk_3de3909e-eda5-4e92-8bad-16b046ef465e/revoke-keys/0.log" Oct 30 22:42:53.119978 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:53.119949 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-5tqnp_ad82aae6-0e74-43d0-afc9-0232c6c69ff2/revoke-keys/0.log" Oct 30 22:42:53.323056 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:53.323035 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-glz68_134c7d44-e7a3-4463-9436-1c93da07e6b5/revoke-keys/0.log" Oct 30 22:42:54.623612 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:54.623582 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-676b586c44-xvzr8_91184aa4-0bba-49fb-a9a7-c3fca31ce6e7/authorino/0.log" Oct 30 22:42:55.073026 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:55.072999 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zj45z_00f89aea-4186-454a-a88b-91ac4a55b503/manager/0.log" Oct 30 22:42:55.298449 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:55.298416 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tbpwm_c1b48785-35d1-41cf-b032-b74d4d74bfee/manager/0.log" Oct 30 22:42:55.744353 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:55.744330 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6prnn_880c49b5-0c92-43df-8e8d-091d4d1e19cc/manager/0.log" Oct 30 22:42:56.013078 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:56.013006 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h5bzd_4d577a18-2871-4312-97f6-8b1e7ecd1fb8/limitador/0.log" Oct 30 22:42:56.240340 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:56.240308 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zgqr7_2a86402f-2067-40c8-a53f-fe523aaebebc/manager/0.log" Oct 30 22:42:56.822343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:56.822300 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d97d4qh_8ec5f880-07fb-4331-ad83-e193ebbf62f2/istio-proxy/0.log" Oct 30 22:42:57.679340 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:57.679314 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-2877x_fec7a677-1d2b-416f-8de9-0a65229fd10d/istio-proxy/0.log" Oct 30 22:42:58.130250 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:58.130217 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-4s928_3a5021c0-a209-41c7-8416-f72d83edc4a1/istio-proxy/0.log" Oct 30 22:42:58.543338 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:42:58.543301 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:42:58.597307 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:58.597283 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-g6pnq_acef2b25-5c28-44db-8c2b-ba32a12ddce6/istio-proxy/1.log" Oct 30 22:42:59.980905 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:42:59.980873 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-btd7k_e7a612dc-b817-4e16-90c3-a73530490ad9/istio-proxy/0.log" Oct 30 22:43:00.608992 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:43:00.608965 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7444c59677-nrb6q_3be119ed-6228-46e8-a71b-8f7fb6e812f8/router/0.log" Oct 30 22:43:28.542776 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:43:28.542739 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:43:58.542863 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:43:58.542822 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:44:22.491420 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:22.491387 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qdn6l_5214b527-3150-40e4-9162-f712d602c83a/global-pull-secret-syncer/0.log" Oct 30 22:44:23.082530 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:23.082482 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dj4nw_7dd70011-6085-49a8-b665-cb33157874fe/connection-checker/0.log" Oct 30 22:44:23.508145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:23.508076 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ptmmz_f8ea0e11-f736-4db0-9ffe-d46ed7479dd9/connection-checker/0.log" Oct 30 22:44:23.929350 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:23.929325 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-tvk4h_b10b2b7a-ecaf-434a-8248-0bdfd7df1667/connection-checker/0.log" Oct 30 22:44:25.320508 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:25.320473 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v4lbg_2e5d51c9-d860-45d8-b9c0-6e11fdd73ca9/konnectivity-agent/0.log" Oct 30 22:44:25.741284 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:25.741207 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-250.us-west-1.compute.internal_801ebfba1db02fef7c7a04b1978809fc/haproxy/0.log" Oct 30 22:44:28.543269 ip-10-0-131-250 kubenswrapper[4933]: E1030 22:44:28.543230 4933 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m2qts" containerName="extractor" Oct 30 22:44:31.321373 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.321340 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-676b586c44-xvzr8_91184aa4-0bba-49fb-a9a7-c3fca31ce6e7/authorino/0.log" Oct 30 22:44:31.500571 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.500543 4933 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6"] Oct 30 22:44:31.500949 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.500938 4933 cpu_manager.go:414] "containerMap: removing container" podUID="ad82aae6-0e74-43d0-afc9-0232c6c69ff2" containerName="revoke-keys" Oct 30 22:44:31.500994 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.500951 4933 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad82aae6-0e74-43d0-afc9-0232c6c69ff2" containerName="revoke-keys" Oct 30 22:44:31.501030 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.501013 4933 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad82aae6-0e74-43d0-afc9-0232c6c69ff2" containerName="revoke-keys" Oct 30 22:44:31.504145 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.504129 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.558144 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.558119 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mff8v\"/\"openshift-service-ca.crt\"" Oct 30 22:44:31.609150 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.609094 4933 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mff8v\"/\"kube-root-ca.crt\"" Oct 30 22:44:31.609258 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.609186 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mff8v\"/\"default-dockercfg-nl99d\"" Oct 30 22:44:31.611485 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.611466 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6"] Oct 30 22:44:31.652334 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.652301 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-lib-modules\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.652441 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.652344 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-podres\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.652441 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.652374 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-proc\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.652560 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.652548 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98rd\" (UniqueName: \"kubernetes.io/projected/04eafd53-f396-49d9-b190-c53f1ea57dde-kube-api-access-r98rd\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.652599 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.652581 4933 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-sys\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.679214 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.679194 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zj45z_00f89aea-4186-454a-a88b-91ac4a55b503/manager/0.log" Oct 30 22:44:31.753251 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753231 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98rd\" (UniqueName: \"kubernetes.io/projected/04eafd53-f396-49d9-b190-c53f1ea57dde-kube-api-access-r98rd\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753350 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753265 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-sys\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753350 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753310 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-lib-modules\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753350 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753341 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-podres\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753367 4933 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-proc\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753367 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-sys\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753422 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-lib-modules\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753438 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-podres\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.753467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.753440 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04eafd53-f396-49d9-b190-c53f1ea57dde-proc\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:31.819999 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:31.819972 4933 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98rd\" (UniqueName: \"kubernetes.io/projected/04eafd53-f396-49d9-b190-c53f1ea57dde-kube-api-access-r98rd\") pod \"perf-node-gather-daemonset-f79h6\" (UID: \"04eafd53-f396-49d9-b190-c53f1ea57dde\") " pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:32.120188 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.120162 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tbpwm_c1b48785-35d1-41cf-b032-b74d4d74bfee/manager/0.log" Oct 30 22:44:32.168123 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.168103 4933 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mff8v\"/\"default-dockercfg-nl99d\"" Oct 30 22:44:32.176689 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.176668 4933 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:32.568066 ip-10-0-131-250 kubenswrapper[4933]: W1030 22:44:32.568032 4933 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04eafd53_f396_49d9_b190_c53f1ea57dde.slice/crio-225838d7428ef5f82d2821a39875740289f1ab8c64395ca214992d2918a1d856 WatchSource:0}: Error finding container 225838d7428ef5f82d2821a39875740289f1ab8c64395ca214992d2918a1d856: Status 404 returned error can't find the container with id 225838d7428ef5f82d2821a39875740289f1ab8c64395ca214992d2918a1d856 Oct 30 22:44:32.569688 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.569653 4933 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6"] Oct 30 22:44:32.782343 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.782308 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" event={"ID":"04eafd53-f396-49d9-b190-c53f1ea57dde","Type":"ContainerStarted","Data":"fe92c946a76c1ec57e11972429d35ca6ba7b27d58dbf944bb292e5266dc32022"} Oct 30 22:44:32.782467 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.782366 4933 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" event={"ID":"04eafd53-f396-49d9-b190-c53f1ea57dde","Type":"ContainerStarted","Data":"225838d7428ef5f82d2821a39875740289f1ab8c64395ca214992d2918a1d856"} Oct 30 22:44:32.782543 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.782517 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:32.900123 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:32.900040 4933 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" podStartSLOduration=1.9000275439999998 podStartE2EDuration="1.900027544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:32.89788498 +0000 UTC m=+3721.579034883" watchObservedRunningTime="2026-10-30 22:44:32.900027544 +0000 UTC m=+3721.581177436" Oct 30 22:44:33.035585 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:33.035548 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6prnn_880c49b5-0c92-43df-8e8d-091d4d1e19cc/manager/0.log" Oct 30 22:44:33.415867 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:33.415839 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h5bzd_4d577a18-2871-4312-97f6-8b1e7ecd1fb8/limitador/0.log" Oct 30 22:44:33.894706 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:33.894678 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-zgqr7_2a86402f-2067-40c8-a53f-fe523aaebebc/manager/0.log" Oct 30 22:44:38.798563 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:38.798534 4933 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mff8v/perf-node-gather-daemonset-f79h6" Oct 30 22:44:39.505714 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:39.505686 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-ljcqn_8b8dc362-93da-466a-b4bd-7d33a25a00e4/cluster-monitoring-operator/0.log" Oct 30 22:44:41.753057 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:41.752990 4933 scope.go:122] "RemoveContainer" containerID="ec96133cf93bb3031a32d2c00472f0f2658f6c3205e66e0a8a2b11d01ebb9681" Oct 30 22:44:44.631547 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:44.631517 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4m48_b2392a57-633c-42a4-92e5-64fbb91ce9fc/node-exporter/0.log" Oct 30 22:44:45.046337 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:45.046310 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4m48_b2392a57-633c-42a4-92e5-64fbb91ce9fc/kube-rbac-proxy/0.log" Oct 30 22:44:45.461551 ip-10-0-131-250 kubenswrapper[4933]: I1030 22:44:45.461452 4933 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z4m48_b2392a57-633c-42a4-92e5-64fbb91ce9fc/init-textfile/0.log"